2024-03-25T17:51:18.150 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T17:51:18.150 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312 --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312/orig.config.yaml 2024-03-25T17:51:18.191 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:51:18.263 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621312' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSVD7vP2KGEGEkXf8ZFIHPD1QPwQEUOKMkbsOPXii1QG6h3uRKB2zvn8oiBAn+z+4YhpYIGHVnkHe0aNLvWdS0= smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUmqGh/fvs4G4Grt3FCYf2UIBDkVXoq+MdCOmetXyIl4A1NWCCJbh5kiJQcDb5hCwB04Fl8GbhBTVyZKB+7Tdg= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T17:51:18.263 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T17:51:18.264 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T17:51:18.264 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T17:51:18.265 INFO:teuthology.task.internal:Checking packages... 2024-03-25T17:51:18.288 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T17:51:18.289 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T17:51:18.289 INFO:teuthology.packaging:ref: None 2024-03-25T17:51:18.289 INFO:teuthology.packaging:tag: None 2024-03-25T17:51:18.289 INFO:teuthology.packaging:branch: main 2024-03-25T17:51:18.289 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:51:18.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T17:51:18.628 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T17:51:18.629 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T17:51:18.636 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T17:51:18.636 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T17:51:18.700 INFO:teuthology.task.internal:Saving configuration 2024-03-25T17:51:18.715 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T17:51:18.722 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:51:18.746 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:12.706049', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSVD7vP2KGEGEkXf8ZFIHPD1QPwQEUOKMkbsOPXii1QG6h3uRKB2zvn8oiBAn+z+4YhpYIGHVnkHe0aNLvWdS0='} 2024-03-25T17:51:18.767 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:12.704349', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUmqGh/fvs4G4Grt3FCYf2UIBDkVXoq+MdCOmetXyIl4A1NWCCJbh5kiJQcDb5hCwB04Fl8GbhBTVyZKB+7Tdg='} 2024-03-25T17:51:18.767 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T17:51:18.773 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T17:51:18.773 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T17:51:18.773 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T17:51:18.853 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9d871530d0>, signals=[15]) 2024-03-25T17:51:18.853 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T17:51:18.862 INFO:teuthology.task.internal:Opening connections... 2024-03-25T17:51:18.862 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T17:51:18.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:51:18.954 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T17:51:18.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:51:19.042 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T17:51:19.054 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-03-25T17:51:19.072 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-03-25T17:51:19.072 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-03-25T17:51:19.129 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2024-03-25T17:51:19.129 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="9" 2024-03-25T17:51:19.130 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2024-03-25T17:51:19.130 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:51:19.130 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="9" 2024-03-25T17:51:19.130 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:51:19.130 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:51:19.130 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2024-03-25T17:51:19.130 INFO:teuthology.orchestra.run.smithi123.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:51:19.130 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:51:19.130 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:51:19.130 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:51:19.131 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:51:19.131 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:51:19.131 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-03-25T17:51:19.155 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-03-25T17:51:19.173 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-03-25T17:51:19.173 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-03-25T17:51:19.230 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-03-25T17:51:19.230 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-03-25T17:51:19.230 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-03-25T17:51:19.230 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:51:19.230 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-03-25T17:51:19.230 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:51:19.230 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:51:19.230 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-03-25T17:51:19.231 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:51:19.231 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:51:19.231 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:51:19.231 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:51:19.231 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:51:19.231 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:51:19.231 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-03-25T17:51:19.255 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T17:51:19.268 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T17:51:19.297 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T17:51:19.297 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:51:19.299 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:51:19.315 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T17:51:19.328 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T17:51:19.329 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:51:19.356 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:51:19.461 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T17:51:19.515 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T17:51:19.530 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T17:51:19.531 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T17:51:19.531 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:51:19.531 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:51:19.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:19.547 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-25T17:51:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:19.548 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-03-25T17:51:19.603 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:51:19.603 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-361.el9.x86_64 2024-03-25T17:51:19.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-03-25T17:51:19.605 INFO:teuthology.orchestra.run.smithi123.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:51:19.605 INFO:teuthology.task.kernel:Running kernel on smithi123: 5.14.0-361.el9.x86_64 2024-03-25T17:51:19.605 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum install -y kernel 2024-03-25T17:51:21.738 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T17:51:21.910 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T17:51:23.397 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-25T17:51:23.552 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T17:51:29.399 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 1.5 MB/s | 8.0 MB 00:05 2024-03-25T17:51:31.285 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.0 MB 00:07 2024-03-25T17:51:37.554 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 19 MB 00:07 2024-03-25T17:51:40.049 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 19 MB 00:07 2024-03-25T17:51:47.623 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 953 kB/s | 6.1 MB 00:06 2024-03-25T17:51:49.877 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T17:51:52.104 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-03-25T17:51:53.602 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 624 kB/s | 6.1 MB 00:10 2024-03-25T17:51:54.957 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-03-25T17:51:57.231 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 6.1 kB/s | 1.7 kB 00:00 2024-03-25T17:51:57.316 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 21 MB 00:02 2024-03-25T17:51:59.344 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:51:59.344 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:51:59.442 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T17:51:59.457 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T17:51:59.457 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-25T17:51:59.457 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T17:51:59.457 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T17:51:59.457 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:51:59.457 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T17:51:59.457 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:51:59.458 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:51:59.458 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:51:59.458 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T17:51:59.458 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T17:51:59.458 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T17:51:59.458 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-03-25T17:51:59.458 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T17:51:59.458 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 76 M 2024-03-25T17:51:59.459 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 123 M 2024-03-25T17:51:59.459 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T17:51:59.968 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:59.969 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:59.986 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:59.987 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:59.989 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:00.006 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:00.147 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:00.148 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:00.166 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:00.537 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:00.565 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:00.582 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:00.610 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:00.633 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:00.669 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:00.751 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:00.785 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:00.792 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:00.842 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:00.843 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:00.888 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:00.889 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:00.911 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:00.935 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:00.957 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:01.017 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:01.037 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:01.039 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:01.056 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:01.063 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:01.103 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:01.122 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:01.134 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:01.282 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:01.301 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:01.308 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:01.336 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:01.344 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:01.377 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:01.377 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:01.411 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:01.430 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:01.925 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:01.964 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:01.985 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:02.174 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T17:52:02.253 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:02.275 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:02.330 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:02.442 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:02.444 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:02.461 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:02.499 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:02.536 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:02.636 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:02.671 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:02.672 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:02.689 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:02.822 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:02.822 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:02.856 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:02.856 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:02.874 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:02.874 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:02.875 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:02.916 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:02.926 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:02.968 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:03.082 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:03.105 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:03.172 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:03.188 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:03.227 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:03.261 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:03.354 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 19 kB/s | 66 kB 00:03 2024-03-25T17:52:03.523 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:03.649 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:04.172 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.7 MB/s | 33 MB 00:04 2024-03-25T17:52:04.280 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:52:04.281 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:52:04.322 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.3 MB/s | 15 MB 00:04 2024-03-25T17:52:04.370 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T17:52:04.378 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T17:52:04.379 INFO:teuthology.orchestra.run.smithi123.stdout:Install 4 Packages 2024-03-25T17:52:04.380 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T17:52:04.380 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 76 M 2024-03-25T17:52:04.380 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 123 M 2024-03-25T17:52:04.380 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T17:52:04.402 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:04.475 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:04.476 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:04.476 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:04.562 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:04.588 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:04.590 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:04.607 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:04.693 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:04.749 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:04.754 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:04.759 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:04.768 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:04.849 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:04.891 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:04.988 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.035 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.095 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:05.121 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:05.130 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:05.144 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:05.147 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:05.155 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:05.180 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:05.183 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.187 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:05.220 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:05.221 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:05.253 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:05.272 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:05.302 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.357 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:05.374 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:05.462 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.483 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.505 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.509 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.519 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.525 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.555 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.565 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.570 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.667 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.671 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.698 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.730 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:05.739 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.743 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.778 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.934 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.938 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.959 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:06.001 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:06.007 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:06.025 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:06.138 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:06.152 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:06.174 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:06.258 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:06.336 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:06.431 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:06.465 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:06.499 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:06.653 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:06.698 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:06.699 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:06.753 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:06.896 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:07.009 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:07.036 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:07.043 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:07.077 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:07.078 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:07.112 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:07.112 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:07.130 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:07.130 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:07.133 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:07.169 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:07.169 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:07.221 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:07.271 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:07.306 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:07.327 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:07.341 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:07.426 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:07.459 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:07.552 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:07.648 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:07.678 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:07.783 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:07.849 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:07.854 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:07.871 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:07.958 INFO:teuthology.orchestra.run.smithi123.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 19 kB/s | 66 kB 00:03 2024-03-25T17:52:07.959 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:07.959 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:08.043 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:08.084 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:08.167 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.7 MB/s | 27 MB 00:04 2024-03-25T17:52:08.174 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:08.174 INFO:teuthology.orchestra.run.smithi151.stdout:Total 8.7 MB/s | 76 MB 00:08 2024-03-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:08.683 INFO:teuthology.orchestra.run.smithi123.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.5 MB/s | 15 MB 00:04 2024-03-25T17:52:08.784 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:08.861 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T17:52:09.134 INFO:teuthology.orchestra.run.smithi123.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.1 MB/s | 33 MB 00:04 2024-03-25T17:52:09.134 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:09.170 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:09.210 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T17:52:09.210 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T17:52:09.337 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:09.392 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:09.481 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:09.527 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:09.592 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:09.682 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:09.749 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:09.873 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:10.223 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T17:52:10.223 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T17:52:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:10.484 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:10.543 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:10.877 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:10.897 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:10.914 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:10.998 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:11.242 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:12.042 INFO:teuthology.orchestra.run.smithi123.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 6.7 MB/s | 27 MB 00:04 2024-03-25T17:52:12.047 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:12.047 INFO:teuthology.orchestra.run.smithi123.stdout:Total 9.9 MB/s | 76 MB 00:07 2024-03-25T17:52:12.703 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T17:52:12.710 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T17:52:13.070 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T17:52:13.070 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T17:52:13.348 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:13.430 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:14.093 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T17:52:14.094 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T17:52:16.185 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:16.315 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:16.578 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T17:52:17.185 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:17.267 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:20.024 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:20.162 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:25.019 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:25.909 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:29.719 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:55.271 INFO:teuthology.orchestra.run.smithi151.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:52:55.271 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T17:52:55.315 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:55.656 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:55.656 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:55.656 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:55.656 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:56.108 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:56.109 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T17:52:56.109 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T17:52:56.109 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:56.109 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:56.109 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:52:56.109 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:56.109 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T17:52:56.109 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T17:52:56.448 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:52:56.559 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:56.559 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:52:56.559 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:52:56.559 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:52:56.559 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:52:56.560 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T17:52:56.560 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:52:56.560 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:52:56.640 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:56.641 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:52:56.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo rpm -qi grub2-tools 2024-03-25T17:52:56.675 INFO:teuthology.orchestra.run.smithi151.stdout:Name : grub2-tools 2024-03-25T17:52:56.675 INFO:teuthology.orchestra.run.smithi151.stdout:Epoch : 1 2024-03-25T17:52:56.675 INFO:teuthology.orchestra.run.smithi151.stdout:Version : 2.06 2024-03-25T17:52:56.675 INFO:teuthology.orchestra.run.smithi151.stdout:Release : 68.el9 2024-03-25T17:52:56.675 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: x86_64 2024-03-25T17:52:56.675 INFO:teuthology.orchestra.run.smithi151.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:52:56.675 INFO:teuthology.orchestra.run.smithi151.stdout:Group : Unspecified 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:Size : 8264254 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:License : GPLv3+ 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:Packager : builder@centos.org 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:Vendor : CentOS 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:Summary : Support tools for GRUB. 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:Description : 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T17:52:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:52:56.677 INFO:teuthology.orchestra.run.smithi151.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:52:56.677 INFO:teuthology.orchestra.run.smithi151.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:52:56.677 INFO:teuthology.orchestra.run.smithi151.stdout:hardware devices. 2024-03-25T17:52:56.677 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T17:52:56.677 INFO:teuthology.orchestra.run.smithi151.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:52:56.678 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:52:56.678 INFO:teuthology.task.kernel:Updating grub on smithi151 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:52:56.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:52:57.276 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-03-25T17:52:58.604 INFO:teuthology.orchestra.run.smithi151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:52:58.666 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-03-25T17:52:58.668 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-03-25T17:52:58.685 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.rfaQBU41Qw 2024-03-25T17:52:58.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rfaQBU41Qw 2024-03-25T17:52:58.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.rfaQBU41Qw 2024-03-25T17:52:58.820 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:58.820 INFO:teuthology.orchestra.run.smithi123.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:52:58.820 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T17:52:58.864 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:58.893 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.rfaQBU41Qw is 6KB 2024-03-25T17:52:58.941 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.rfaQBU41Qw 2024-03-25T17:52:58.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:52:59.020 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:52:59.020 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:52:59.020 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:52:59.021 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:52:59.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:52:59.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2024-03-25T17:52:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T17:52:59.585 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T17:52:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:52:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T17:52:59.585 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T17:52:59.914 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:53:00.016 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:00.017 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:53:00.017 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:53:00.017 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:53:00.017 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:53:00.017 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T17:53:00.017 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:53:00.017 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:53:00.096 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:00.097 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:53:00.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2024-03-25T17:53:00.136 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2024-03-25T17:53:00.136 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2024-03-25T17:53:00.136 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.06 2024-03-25T17:53:00.136 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 68.el9 2024-03-25T17:53:00.136 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2024-03-25T17:53:00.136 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:53:00.136 INFO:teuthology.orchestra.run.smithi123.stdout:Group : Unspecified 2024-03-25T17:53:00.136 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 8264254 2024-03-25T17:53:00.136 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2024-03-25T17:53:00.137 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:53:00.137 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:53:00.137 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:53:00.137 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:53:00.137 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : builder@centos.org 2024-03-25T17:53:00.137 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2024-03-25T17:53:00.137 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:53:00.137 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2024-03-25T17:53:00.137 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2024-03-25T17:53:00.137 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T17:53:00.138 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:53:00.138 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:53:00.138 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:53:00.138 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2024-03-25T17:53:00.138 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T17:53:00.138 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:53:00.139 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:53:00.139 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:53:00.139 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:53:00.712 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-03-25T17:53:02.031 INFO:teuthology.orchestra.run.smithi123.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:53:02.061 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-03-25T17:53:02.063 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-03-25T17:53:02.078 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.OGGizTfZQ9 2024-03-25T17:53:02.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OGGizTfZQ9 2024-03-25T17:53:02.147 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.OGGizTfZQ9 2024-03-25T17:53:02.284 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.OGGizTfZQ9 is 6KB 2024-03-25T17:53:02.332 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.OGGizTfZQ9 2024-03-25T17:53:02.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:53:02.411 INFO:teuthology.orchestra.run.smithi123.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:53:02.411 INFO:teuthology.orchestra.run.smithi123.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:53:02.411 INFO:teuthology.orchestra.run.smithi123.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:53:02.411 INFO:teuthology.orchestra.run.smithi123.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:53:02.412 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:53:02.566 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-03-25T17:53:29.196 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T17:53:29.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-25T17:53:29.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:53:32.569 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T17:53:32.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-25T17:53:32.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:03.679 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-25T17:54:12.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-25T17:54:12.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:32.590 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:54:41.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-25T17:54:41.601 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:43.039 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-03-25T17:54:43.559 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-25T17:54:43.559 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:54:43.559 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-03-25T17:54:43.573 INFO:teuthology.orchestra.run.smithi123.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:54:43.573 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:54:43.573 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:54:43.573 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:54:44.575 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T17:54:44.575 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:54:44.637 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2024-03-25T17:54:44.677 DEBUG:teuthology.parallel:result is None 2024-03-25T17:55:12.692 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:55:24.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-25T17:55:24.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:55:25.042 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-25T17:55:25.441 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-25T17:55:25.441 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:55:25.441 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-25T17:55:25.457 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:55:25.457 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:55:25.457 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:55:25.457 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:55:26.458 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T17:55:26.459 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:55:26.534 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-03-25T17:55:26.565 DEBUG:teuthology.parallel:result is None 2024-03-25T17:55:26.565 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T17:55:26.575 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T17:55:26.575 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:55:26.578 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:55:26.596 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T17:55:26.603 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T17:55:26.622 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T17:55:26.622 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:55:26.637 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:55:26.665 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T17:55:26.675 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T17:55:26.676 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:55:26.699 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:55:26.736 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:26.744 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:26.750 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:26.758 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:26.760 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T17:55:26.771 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T17:55:26.771 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:55:26.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:55:26.824 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T17:55:26.835 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T17:55:26.835 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:55:26.863 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:55:26.881 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:26.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:27.007 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:27.088 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:27.131 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T17:55:27.131 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:55:27.197 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:27.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:27.297 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:27.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:27.426 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T17:55:27.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:55:27.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-03-25T17:55:27.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-03-25T17:55:27.543 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:55:27.583 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:55:27.975 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T17:55:27.984 INFO:teuthology.task.internal:Starting timer... 2024-03-25T17:55:27.984 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T17:55:28.023 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T17:55:28.040 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2024-03-25T17:55:28.102 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2024-03-25T17:55:28.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-03-25T17:55:28.163 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-03-25T17:55:28.165 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T17:55:28.166 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2024-03-25T17:55:28.184 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2024-03-25T17:55:28.185 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-03-25T17:55:28.208 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-03-25T17:55:28.208 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi123.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi151.front.sepia.ceph.com': 'permissive'} 2024-03-25T17:55:28.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:55:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:55:28.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:55:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:55:28.281 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T17:55:28.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:55:28.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:55:28.359 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T17:55:28.368 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T17:55:28.389 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T17:55:28.391 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi123.front.sepia.ceph.com,smithi151.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T17:59:51.580 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi151.front.sepia.ceph.com')] 2024-03-25T17:59:51.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-25T17:59:51.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:59:51.677 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-03-25T17:59:51.753 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-25T17:59:51.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-25T17:59:51.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:59:51.845 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-25T17:59:51.922 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-25T17:59:51.922 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T17:59:51.936 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T17:59:51.936 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:59:51.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:59:51.938 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:59:51.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:59:51.967 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:59:51.984 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:59:51.994 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:59:52.011 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:59:52.012 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2024-03-25T17:59:52.026 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2024-03-25T17:59:52.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-03-25T17:59:52.043 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:59:52.054 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-03-25T17:59:52.059 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:59:52.070 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:59:52.087 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:59:52.117 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:59:52.120 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:59:52.120 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-03-25T17:59:52.120 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:52.120 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:52.120 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:52.121 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:52.140 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:59:52.143 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:59:52.143 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-25T17:59:52.143 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:52.144 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:52.144 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:52.144 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:52.145 INFO:teuthology.run_tasks:Running task install... 2024-03-25T17:59:52.153 DEBUG:teuthology.task.install:project ceph 2024-03-25T17:59:52.153 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T17:59:52.153 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T17:59:52.153 INFO:teuthology.task.install:Using flavor: default 2024-03-25T17:59:52.158 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T17:59:52.158 INFO:teuthology.task.install:extra packages: [] 2024-03-25T17:59:52.158 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:59:52.159 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:59:52.161 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:59:52.161 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:59:52.375 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:59:52.375 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:59:52.384 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:59:52.384 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:59:52.715 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:59:52.716 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T17:59:52.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:59:52.746 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:59:52.746 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:59:52.788 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:59:52.788 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T17:59:52.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:59:52.818 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:59:52.818 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:59:52.819 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:59:52.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:59:52.902 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:59:52.969 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:59:52.969 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2024-03-25T17:59:52.971 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2024-03-25T17:59:53.036 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2024-03-25T17:59:53.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2024-03-25T17:59:53.216 INFO:teuthology.orchestra.run.smithi123.stdout:74 files removed 2024-03-25T17:59:53.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw 2024-03-25T17:59:53.289 INFO:teuthology.orchestra.run.smithi151.stdout:74 files removed 2024-03-25T17:59:53.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw 2024-03-25T17:59:54.492 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 86 kB/s | 82 kB 00:00 2024-03-25T17:59:54.966 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 61 kB/s | 82 kB 00:01 2024-03-25T17:59:55.318 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T17:59:55.825 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T17:59:56.152 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T17:59:56.629 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T17:59:56.735 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T17:59:57.311 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T17:59:58.021 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-25T17:59:58.978 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-25T18:00:07.019 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 951 kB/s | 8.0 MB 00:08 2024-03-25T18:00:07.742 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 977 kB/s | 8.0 MB 00:08 2024-03-25T18:00:17.854 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 19 MB 00:09 2024-03-25T18:00:18.645 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 19 MB 00:09 2024-03-25T18:00:26.933 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 6.1 MB 00:05 2024-03-25T18:00:27.907 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 6.1 MB 00:05 2024-03-25T18:00:28.371 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-25T18:00:29.720 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-25T18:00:30.140 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T18:00:31.183 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T18:00:35.014 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-25T18:00:36.113 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-25T18:00:37.458 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:00:37.474 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:00:37.474 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-25T18:00:37.474 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:00:37.474 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:00:37.474 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:00:37.474 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-03-25T18:00:37.474 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:00:37.474 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:00:37.475 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:00:37.476 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout:Install 30 Packages 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 2 Packages 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 58 M 2024-03-25T18:00:37.477 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:00:38.579 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:00:38.594 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:00:38.594 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-25T18:00:38.594 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:00:38.594 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:00:38.594 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:00:38.594 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:00:38.595 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:00:38.596 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout:Installing weak dependencies: 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout:Install 30 Packages 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 2 Packages 2024-03-25T18:00:38.597 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:00:38.598 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 58 M 2024-03-25T18:00:38.598 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:00:53.889 INFO:teuthology.orchestra.run.smithi123.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 338 kB/s | 5.3 MB 00:16 2024-03-25T18:00:53.972 INFO:teuthology.orchestra.run.smithi123.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 303 kB/s | 25 kB 00:00 2024-03-25T18:00:55.447 INFO:teuthology.orchestra.run.smithi123.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 503 kB/s | 742 kB 00:01 2024-03-25T18:00:56.523 INFO:teuthology.orchestra.run.smithi123.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 464 kB/s | 498 kB 00:01 2024-03-25T18:01:02.546 INFO:teuthology.orchestra.run.smithi151.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 231 kB/s | 5.3 MB 00:23 2024-03-25T18:01:02.662 INFO:teuthology.orchestra.run.smithi151.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T18:01:05.713 INFO:teuthology.orchestra.run.smithi151.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 243 kB/s | 742 kB 00:03 2024-03-25T18:01:08.429 INFO:teuthology.orchestra.run.smithi151.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 183 kB/s | 498 kB 00:02 2024-03-25T18:01:10.763 INFO:teuthology.orchestra.run.smithi123.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 367 kB/s | 5.1 MB 00:14 2024-03-25T18:01:10.975 INFO:teuthology.orchestra.run.smithi123.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 213 kB/s | 45 kB 00:00 2024-03-25T18:01:11.805 INFO:teuthology.orchestra.run.smithi123.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 158 kB/s | 131 kB 00:00 2024-03-25T18:01:12.764 INFO:teuthology.orchestra.run.smithi123.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 169 kB/s | 162 kB 00:00 2024-03-25T18:01:13.964 INFO:teuthology.orchestra.run.smithi123.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 267 kB/s | 320 kB 00:01 2024-03-25T18:01:15.014 INFO:teuthology.orchestra.run.smithi123.stdout:(10/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 267 kB/s | 9.7 MB 00:37 2024-03-25T18:01:15.039 INFO:teuthology.orchestra.run.smithi123.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 277 kB/s | 298 kB 00:01 2024-03-25T18:01:15.148 INFO:teuthology.orchestra.run.smithi123.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 980 kB/s | 106 kB 00:00 2024-03-25T18:01:15.190 INFO:teuthology.orchestra.run.smithi123.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T18:01:15.223 INFO:teuthology.orchestra.run.smithi123.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:01:15.273 INFO:teuthology.orchestra.run.smithi123.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-25T18:01:15.315 INFO:teuthology.orchestra.run.smithi123.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:01:15.349 INFO:teuthology.orchestra.run.smithi123.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:01:15.449 INFO:teuthology.orchestra.run.smithi123.stdout:(18/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 228 kB/s | 99 kB 00:00 2024-03-25T18:01:15.529 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:15.635 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:15.886 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:15.921 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:15.962 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:16.010 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:16.026 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:16.073 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:16.074 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:16.093 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:16.201 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:16.219 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:16.221 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:16.238 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:16.694 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:16.718 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:17.034 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:17.047 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:17.104 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:17.119 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:17.243 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:17.265 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:17.349 INFO:teuthology.orchestra.run.smithi123.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 19 kB/s | 36 kB 00:01 2024-03-25T18:01:17.383 INFO:teuthology.orchestra.run.smithi123.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 109 kB/s | 221 kB 00:02 2024-03-25T18:01:17.511 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:17.641 INFO:teuthology.orchestra.run.smithi123.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-25T18:01:17.861 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:17.895 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:17.967 INFO:teuthology.orchestra.run.smithi123.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-25T18:01:17.988 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:18.017 INFO:teuthology.orchestra.run.smithi123.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-03-25T18:01:18.017 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:18.059 INFO:teuthology.orchestra.run.smithi123.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-25T18:01:18.078 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:18.109 INFO:teuthology.orchestra.run.smithi123.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-25T18:01:18.109 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:18.218 INFO:teuthology.orchestra.run.smithi123.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-03-25T18:01:18.268 INFO:teuthology.orchestra.run.smithi123.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-03-25T18:01:18.452 INFO:teuthology.orchestra.run.smithi123.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-25T18:01:18.618 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:18.954 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:18.987 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:19.019 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:19.052 INFO:teuthology.orchestra.run.smithi123.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 28 kB/s | 47 kB 00:01 2024-03-25T18:01:22.536 INFO:teuthology.orchestra.run.smithi151.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 228 kB/s | 9.7 MB 00:43 2024-03-25T18:01:22.636 INFO:teuthology.orchestra.run.smithi151.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-25T18:01:22.978 INFO:teuthology.orchestra.run.smithi151.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 384 kB/s | 131 kB 00:00 2024-03-25T18:01:23.553 INFO:teuthology.orchestra.run.smithi151.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 282 kB/s | 162 kB 00:00 2024-03-25T18:01:24.927 INFO:teuthology.orchestra.run.smithi123.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 545 kB/s | 3.1 MB 00:05 2024-03-25T18:01:24.962 INFO:teuthology.orchestra.run.smithi151.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 228 kB/s | 320 kB 00:01 2024-03-25T18:01:26.479 INFO:teuthology.orchestra.run.smithi151.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 196 kB/s | 298 kB 00:01 2024-03-25T18:01:27.004 INFO:teuthology.orchestra.run.smithi151.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 189 kB/s | 99 kB 00:00 2024-03-25T18:01:27.112 INFO:teuthology.orchestra.run.smithi151.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-25T18:01:27.162 INFO:teuthology.orchestra.run.smithi151.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:01:27.196 INFO:teuthology.orchestra.run.smithi151.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:01:27.255 INFO:teuthology.orchestra.run.smithi151.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:01:27.297 INFO:teuthology.orchestra.run.smithi151.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-25T18:01:27.330 INFO:teuthology.orchestra.run.smithi151.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:01:27.491 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:27.846 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:27.863 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:27.918 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:28.064 INFO:teuthology.orchestra.run.smithi151.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 301 kB/s | 221 kB 00:00 2024-03-25T18:01:28.110 INFO:teuthology.orchestra.run.smithi123.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 358 kB/s | 3.4 MB 00:09 2024-03-25T18:01:28.144 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:28.256 INFO:teuthology.orchestra.run.smithi151.stdout:(19/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 263 kB/s | 5.1 MB 00:19 2024-03-25T18:01:28.256 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:28.274 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:28.329 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:28.336 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:28.373 INFO:teuthology.orchestra.run.smithi151.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 116 kB/s | 36 kB 00:00 2024-03-25T18:01:28.416 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:28.433 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:28.505 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:28.548 INFO:teuthology.orchestra.run.smithi151.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 162 kB/s | 47 kB 00:00 2024-03-25T18:01:29.165 INFO:teuthology.orchestra.run.smithi151.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 390 kB/s | 309 kB 00:00 2024-03-25T18:01:29.274 INFO:teuthology.orchestra.run.smithi151.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-25T18:01:29.382 INFO:teuthology.orchestra.run.smithi151.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 449 kB/s | 49 kB 00:00 2024-03-25T18:01:29.491 INFO:teuthology.orchestra.run.smithi151.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-03-25T18:01:30.583 INFO:teuthology.orchestra.run.smithi151.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 768 kB/s | 838 kB 00:01 2024-03-25T18:01:30.850 INFO:teuthology.orchestra.run.smithi151.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 719 kB/s | 192 kB 00:00 2024-03-25T18:01:32.874 INFO:teuthology.orchestra.run.smithi151.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 800 kB/s | 1.6 MB 00:02 2024-03-25T18:01:34.141 INFO:teuthology.orchestra.run.smithi151.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 809 kB/s | 4.4 MB 00:05 2024-03-25T18:01:45.440 INFO:teuthology.orchestra.run.smithi151.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 284 kB/s | 3.1 MB 00:11 2024-03-25T18:01:46.715 INFO:teuthology.orchestra.run.smithi151.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 250 kB/s | 3.4 MB 00:13 2024-03-25T18:01:51.990 INFO:teuthology.orchestra.run.smithi151.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 278 kB/s | 20 MB 01:13 2024-03-25T18:01:51.995 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:51.995 INFO:teuthology.orchestra.run.smithi151.stdout:Total 807 kB/s | 58 MB 01:13 2024-03-25T18:01:52.028 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:01:52.068 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:01:52.068 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:01:52.551 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:01:52.552 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:01:54.083 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:01:54.177 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:01:54.203 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:01:54.236 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:01:54.259 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:01:54.275 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:01:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:01:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:01:54.848 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:01:54.921 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:55.108 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:01:55.139 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:01:55.210 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:01:55.239 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:01:55.283 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:01:55.350 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:01:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:01:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:01:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:01:55.581 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:01:55.626 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:01:55.665 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:01:55.698 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:01:55.726 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:01:55.759 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:01:56.093 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:01:56.122 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:01:56.212 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:01:56.244 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:01:56.283 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:01:56.736 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:01:56.767 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:01:56.830 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:01:56.898 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:01:56.927 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:01:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:59.671 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:59.726 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:01:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:01:59.810 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:01:59.810 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:01:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:01:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:02.813 INFO:teuthology.orchestra.run.smithi123.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 239 kB/s | 20 MB 01:25 2024-03-25T18:02:02.819 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:02.819 INFO:teuthology.orchestra.run.smithi123.stdout:Total 694 kB/s | 58 MB 01:25 2024-03-25T18:02:02.850 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:02:02.887 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:02:02.887 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:02:03.385 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:02:03.385 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:02:04.852 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:02:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:02:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:02:04.993 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:02:05.016 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:02:05.032 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:02:05.216 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:02:05.272 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:02:05.602 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:02:05.661 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:05.752 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:05.839 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:02:05.871 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:02:05.934 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:02:05.962 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:02:06.006 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:02:06.074 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:02:06.098 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:02:06.146 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:02:06.181 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:02:06.289 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:02:06.331 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:02:06.363 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:02:06.388 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:02:06.416 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:02:06.449 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:02:06.774 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:02:06.804 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:02:06.878 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:02:06.917 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:02:06.962 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:02:07.438 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:02:07.465 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:02:07.519 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:02:07.588 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:02:07.618 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:02:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:09.263 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2024-03-25T18:02:09.264 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2024-03-25T18:02:09.264 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2024-03-25T18:02:09.264 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2024-03-25T18:02:09.264 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2024-03-25T18:02:09.264 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /root 2024-03-25T18:02:09.264 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2024-03-25T18:02:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:02:09.786 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:09.831 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:10.189 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:02:10.213 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:02:10.213 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:02:10.213 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:02:10.213 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:02:10.213 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:02:10.213 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:02:10.215 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:02:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:02:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:10.393 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:10.425 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:02:10.525 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:02:10.525 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:02:10.525 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:02:10.525 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:02:10.560 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:11.440 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:11.440 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:02:11.440 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:02:11.440 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:02:11.440 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:02:11.441 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:02:11.442 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:02:11.442 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:02:11.442 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:02:11.442 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:02:11.442 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:02:11.442 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:02:11.442 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:02:11.442 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:02:11.442 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:02:11.442 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:02:11.443 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:02:11.443 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:11.443 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:02:11.443 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:02:12.586 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:02:12.587 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:02:12.588 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:02:12.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test 2024-03-25T18:02:13.497 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:37 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:02:13.648 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:02:13.649 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:02:13.650 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:02:13.651 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-03-25T18:02:13.651 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:02:13.651 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 48 M 2024-03-25T18:02:13.651 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 204 M 2024-03-25T18:02:13.651 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:02:14.066 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 911 kB/s | 217 kB 00:00 2024-03-25T18:02:14.159 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 300 kB 00:00 2024-03-25T18:02:14.209 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-25T18:02:14.276 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-25T18:02:14.743 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 178 kB/s | 163 kB 00:00 2024-03-25T18:02:19.876 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:19.877 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2024-03-25T18:02:19.877 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2024-03-25T18:02:19.877 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2024-03-25T18:02:19.877 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2024-03-25T18:02:19.877 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2024-03-25T18:02:19.877 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /root 2024-03-25T18:02:19.877 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2024-03-25T18:02:19.877 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:02:20.803 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:02:20.828 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:02:20.828 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:02:20.828 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:02:20.828 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:02:20.828 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:02:20.828 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:02:20.830 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:02:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:02:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:22.031 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:22.031 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:02:22.032 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:02:22.033 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:02:22.034 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:22.034 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:02:22.034 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:02:23.104 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:02:23.105 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.106 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.107 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:23.107 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:02:23.107 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:02:23.107 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:02:23.107 INFO:teuthology.orchestra.run.smithi123.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:02:23.107 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:02:23.107 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:02:23.107 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:02:23.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-test 2024-03-25T18:02:23.967 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:49 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:02:24.115 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:02:24.116 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:02:24.116 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-25T18:02:24.116 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2024-03-25T18:02:24.117 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:02:24.118 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 48 M 2024-03-25T18:02:24.118 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 204 M 2024-03-25T18:02:24.118 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:02:24.420 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T18:02:24.479 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-25T18:02:24.512 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T18:02:24.579 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-25T18:02:25.479 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 133 kB/s | 163 kB 00:01 2024-03-25T18:04:16.596 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 394 kB/s | 47 MB 02:02 2024-03-25T18:04:16.597 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:16.598 INFO:teuthology.orchestra.run.smithi151.stdout:Total 401 kB/s | 48 MB 02:02 2024-03-25T18:04:16.609 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:04:16.628 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:04:16.628 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:04:17.011 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:04:17.011 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:04:17.598 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:04:17.679 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:04:17.722 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:04:17.757 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:04:17.805 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:04:17.832 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:04:22.302 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:04:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:04:24.959 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:04:24.959 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:04:24.959 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:04:24.959 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:04:24.959 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:04:24.959 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:04:25.330 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:04:25.330 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:04:25.330 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:04:25.330 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:25.330 INFO:teuthology.orchestra.run.smithi151.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:04:25.330 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:25.331 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:04:25.331 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:04:25.331 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:04:25.331 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:04:25.331 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:04:25.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph 2024-03-25T18:04:26.198 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:50 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:04:26.357 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:04:26.360 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-03-25T18:04:26.360 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-25T18:04:26.360 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-03-25T18:04:26.360 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:04:26.360 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:04:26.360 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T18:04:26.361 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:04:26.361 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:04:26.361 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:04:26.361 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:04:26.361 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:04:26.361 INFO:teuthology.orchestra.run.smithi151.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:04:26.361 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:04:26.361 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:04:26.361 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:04:26.362 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:04:26.362 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:04:26.362 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:04:26.362 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:04:26.362 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:04:26.362 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:04:26.362 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:04:26.362 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:04:26.362 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:04:26.362 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:04:26.363 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:04:26.363 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:04:26.363 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:04:26.363 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:04:26.363 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:04:26.363 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:04:26.363 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:04:26.363 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:04:26.363 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:04:26.363 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout:Install 36 Packages 2024-03-25T18:04:26.364 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:04:26.365 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 30 M 2024-03-25T18:04:26.365 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 107 M 2024-03-25T18:04:26.365 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:04:27.329 INFO:teuthology.orchestra.run.smithi151.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T18:04:30.838 INFO:teuthology.orchestra.run.smithi151.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 386 kB/s | 1.5 MB 00:03 2024-03-25T18:04:37.404 INFO:teuthology.orchestra.run.smithi151.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 207 kB/s | 2.1 MB 00:10 2024-03-25T18:04:39.138 INFO:teuthology.orchestra.run.smithi151.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 143 kB/s | 247 kB 00:01 2024-03-25T18:04:39.254 INFO:teuthology.orchestra.run.smithi151.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:04:39.296 INFO:teuthology.orchestra.run.smithi151.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:04:39.330 INFO:teuthology.orchestra.run.smithi151.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-03-25T18:04:39.364 INFO:teuthology.orchestra.run.smithi151.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:04:39.398 INFO:teuthology.orchestra.run.smithi151.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T18:04:39.440 INFO:teuthology.orchestra.run.smithi151.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T18:04:39.515 INFO:teuthology.orchestra.run.smithi151.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:04:39.557 INFO:teuthology.orchestra.run.smithi151.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T18:04:39.591 INFO:teuthology.orchestra.run.smithi151.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:04:39.625 INFO:teuthology.orchestra.run.smithi151.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.4 MB/s | 187 kB 00:00 2024-03-25T18:04:39.751 INFO:teuthology.orchestra.run.smithi151.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-25T18:04:40.317 INFO:teuthology.orchestra.run.smithi151.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-25T18:04:40.609 INFO:teuthology.orchestra.run.smithi151.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 361 kB/s | 4.7 MB 00:13 2024-03-25T18:04:40.818 INFO:teuthology.orchestra.run.smithi151.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 302 kB/s | 151 kB 00:00 2024-03-25T18:04:40.968 INFO:teuthology.orchestra.run.smithi151.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-25T18:04:40.994 INFO:teuthology.orchestra.run.smithi151.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 113 kB/s | 43 kB 00:00 2024-03-25T18:04:41.127 INFO:teuthology.orchestra.run.smithi151.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-03-25T18:04:41.178 INFO:teuthology.orchestra.run.smithi151.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-25T18:04:41.236 INFO:teuthology.orchestra.run.smithi151.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-25T18:04:41.270 INFO:teuthology.orchestra.run.smithi151.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-03-25T18:04:41.354 INFO:teuthology.orchestra.run.smithi151.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-03-25T18:04:41.379 INFO:teuthology.orchestra.run.smithi151.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 180 kB/s | 20 kB 00:00 2024-03-25T18:04:41.480 INFO:teuthology.orchestra.run.smithi151.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 789 kB/s | 79 kB 00:00 2024-03-25T18:04:41.505 INFO:teuthology.orchestra.run.smithi151.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 307 kB/s | 46 kB 00:00 2024-03-25T18:04:41.605 INFO:teuthology.orchestra.run.smithi151.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-25T18:04:41.631 INFO:teuthology.orchestra.run.smithi151.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 131 kB/s | 16 kB 00:00 2024-03-25T18:04:41.723 INFO:teuthology.orchestra.run.smithi151.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 768 kB/s | 90 kB 00:00 2024-03-25T18:04:41.748 INFO:teuthology.orchestra.run.smithi151.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 305 kB/s | 36 kB 00:00 2024-03-25T18:04:41.832 INFO:teuthology.orchestra.run.smithi151.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-25T18:04:41.924 INFO:teuthology.orchestra.run.smithi151.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-25T18:04:42.057 INFO:teuthology.orchestra.run.smithi151.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.3 MB/s | 427 kB 00:00 2024-03-25T18:04:58.349 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 314 kB/s | 47 MB 02:34 2024-03-25T18:04:58.352 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:58.352 INFO:teuthology.orchestra.run.smithi123.stdout:Total 320 kB/s | 48 MB 02:34 2024-03-25T18:04:58.363 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:04:58.382 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:04:58.382 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:04:58.759 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:04:58.760 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:04:59.348 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:04:59.408 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:04:59.454 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:04:59.493 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:04:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:04:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:05:04.086 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:05:04.161 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:05:06.340 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:05:06.340 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:05:06.340 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:05:06.341 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:05:06.341 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:05:06.341 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:05:06.766 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:05:06.766 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:05:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:05:06.766 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:06.766 INFO:teuthology.orchestra.run.smithi123.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:05:06.766 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:06.766 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:05:06.766 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:05:06.766 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:05:06.766 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:05:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:05:07.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph 2024-03-25T18:05:07.651 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:04:33 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:05:07.812 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:05:07.815 INFO:teuthology.orchestra.run.smithi123.stdout:========================================================================================= 2024-03-25T18:05:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout:========================================================================================= 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:05:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:05:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:05:07.818 INFO:teuthology.orchestra.run.smithi123.stdout:========================================================================================= 2024-03-25T18:05:07.819 INFO:teuthology.orchestra.run.smithi123.stdout:Install 36 Packages 2024-03-25T18:05:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:05:07.819 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 30 M 2024-03-25T18:05:07.819 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 107 M 2024-03-25T18:05:07.819 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:05:08.927 INFO:teuthology.orchestra.run.smithi123.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-25T18:05:09.138 INFO:teuthology.orchestra.run.smithi151.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 449 kB/s | 17 MB 00:38 2024-03-25T18:05:09.143 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:09.144 INFO:teuthology.orchestra.run.smithi151.stdout:Total 710 kB/s | 30 MB 00:42 2024-03-25T18:05:09.167 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:05:09.211 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:05:09.211 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:05:09.616 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:05:09.616 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:05:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:05:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:05:11.042 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:05:11.077 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:05:11.147 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:05:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:05:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:05:11.317 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:05:11.396 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:05:11.437 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:05:11.509 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:05:11.555 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:05:11.613 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:05:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:05:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:05:11.910 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:05:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:05:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:05:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:05:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:05:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:05:12.155 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:05:12.189 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:05:12.229 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:05:12.266 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:05:12.301 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:05:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:05:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:05:12.504 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:05:12.642 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:05:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:05:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:05:12.955 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:05:12.981 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:05:12.981 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:12.981 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:05:12.981 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:05:12.981 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:05:12.981 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:05:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:05:14.511 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:14.511 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:05:14.511 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:05:14.511 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:05:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:14.959 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:05:14.982 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:05:14.983 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:14.983 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:05:14.983 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:05:14.983 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:05:14.983 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:14.993 INFO:teuthology.orchestra.run.smithi123.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 235 kB/s | 1.5 MB 00:06 2024-03-25T18:05:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:05:15.216 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:05:15.216 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:15.216 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:05:15.216 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:05:15.216 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:05:15.217 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:15.282 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:05:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:05:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:05:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:05:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:05:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:05:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:05:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:05:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:05:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:05:18.179 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:05:18.180 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:18.180 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:05:18.180 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:18.180 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:18.180 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:18.180 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:18.180 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:18.180 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:18.180 INFO:teuthology.orchestra.run.smithi151.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:05:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:05:18.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:05:18.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:05:18.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:05:18.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:05:18.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:05:18.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:05:18.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:05:18.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:05:18.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:05:18.182 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:05:18.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:05:18.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:05:18.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:05:18.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:05:18.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:05:18.183 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:05:18.183 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:18.183 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:05:18.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base 2024-03-25T18:05:19.066 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:05:19.162 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:19.214 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:05:19.215 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-25T18:05:19.216 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:05:19.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install cephadm 2024-03-25T18:05:19.776 INFO:teuthology.orchestra.run.smithi123.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 193 kB/s | 2.1 MB 00:11 2024-03-25T18:05:19.837 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:05:19.985 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:05:19.985 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:05:19.985 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 747 k 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 754 k 2024-03-25T18:05:19.986 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:05:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 199 kB/s | 247 kB 00:01 2024-03-25T18:05:21.143 INFO:teuthology.orchestra.run.smithi123.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T18:05:21.185 INFO:teuthology.orchestra.run.smithi123.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:05:21.218 INFO:teuthology.orchestra.run.smithi123.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 951 kB/s | 32 kB 00:00 2024-03-25T18:05:21.252 INFO:teuthology.orchestra.run.smithi123.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:05:21.302 INFO:teuthology.orchestra.run.smithi123.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 775 kB/s | 39 kB 00:00 2024-03-25T18:05:21.344 INFO:teuthology.orchestra.run.smithi123.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:05:21.428 INFO:teuthology.orchestra.run.smithi123.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T18:05:21.478 INFO:teuthology.orchestra.run.smithi123.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-25T18:05:21.520 INFO:teuthology.orchestra.run.smithi123.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi123.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T18:05:21.779 INFO:teuthology.orchestra.run.smithi123.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 600 kB/s | 125 kB 00:00 2024-03-25T18:05:22.037 INFO:teuthology.orchestra.run.smithi123.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-25T18:05:22.180 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 341 kB/s | 747 kB 00:02 2024-03-25T18:05:22.181 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:22.181 INFO:teuthology.orchestra.run.smithi151.stdout:Total 340 kB/s | 747 kB 00:02 2024-03-25T18:05:22.182 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:05:22.187 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:05:22.188 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:05:22.194 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:05:22.194 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:05:22.271 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:05:22.313 INFO:teuthology.orchestra.run.smithi123.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 549 kB/s | 151 kB 00:00 2024-03-25T18:05:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 872 kB/s | 43 kB 00:00 2024-03-25T18:05:22.446 INFO:teuthology.orchestra.run.smithi123.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-25T18:05:22.547 INFO:teuthology.orchestra.run.smithi123.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-25T18:05:22.597 INFO:teuthology.orchestra.run.smithi123.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-25T18:05:22.647 INFO:teuthology.orchestra.run.smithi123.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-03-25T18:05:22.698 INFO:teuthology.orchestra.run.smithi123.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-03-25T18:05:22.702 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:22.748 INFO:teuthology.orchestra.run.smithi123.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 390 kB/s | 19 kB 00:00 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:22.798 INFO:teuthology.orchestra.run.smithi123.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 395 kB/s | 20 kB 00:00 2024-03-25T18:05:22.848 INFO:teuthology.orchestra.run.smithi123.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 927 kB/s | 46 kB 00:00 2024-03-25T18:05:22.907 INFO:teuthology.orchestra.run.smithi123.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T18:05:22.975 INFO:teuthology.orchestra.run.smithi123.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T18:05:23.025 INFO:teuthology.orchestra.run.smithi123.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-03-25T18:05:23.083 INFO:teuthology.orchestra.run.smithi123.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T18:05:23.133 INFO:teuthology.orchestra.run.smithi123.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-03-25T18:05:23.152 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T18:05:23.276 INFO:teuthology.orchestra.run.smithi123.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-25T18:05:23.326 INFO:teuthology.orchestra.run.smithi123.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-25T18:05:23.432 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:23.433 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:23.433 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:05:23.433 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:23.433 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:23.433 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:05:23.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:05:24.226 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:05:24.375 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:05:24.376 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:05:24.376 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:05:24.376 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:05:24.376 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:05:24.376 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:05:24.376 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:24.377 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:05:24.377 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:05:24.377 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:05:24.377 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:24.377 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 145 k 2024-03-25T18:05:24.377 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 436 k 2024-03-25T18:05:24.377 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:05:25.700 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 110 kB/s | 145 kB 00:01 2024-03-25T18:05:25.700 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:25.701 INFO:teuthology.orchestra.run.smithi151.stdout:Total 110 kB/s | 145 kB 00:01 2024-03-25T18:05:25.701 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:05:25.710 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:05:25.710 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:05:25.766 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:05:25.767 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:05:25.912 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:05:25.967 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:05:25.991 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:05:25.992 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:25.992 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:05:25.992 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:26.759 INFO:teuthology.orchestra.run.smithi123.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 269 kB/s | 4.7 MB 00:17 2024-03-25T18:05:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:05:27.021 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:27.022 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:05:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:27.022 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:27.022 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:05:27.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr 2024-03-25T18:05:27.854 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:05:27.952 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:28.004 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:05:28.005 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-25T18:05:28.005 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:05:28.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:05:28.636 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:05:28.786 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:05:28.787 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-03-25T18:05:28.787 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-25T18:05:28.787 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:05:28.788 INFO:teuthology.orchestra.run.smithi151.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:05:28.789 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:05:28.789 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:05:28.789 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:28.789 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:05:28.789 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-03-25T18:05:28.789 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2024-03-25T18:05:28.789 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:28.790 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 6.9 M 2024-03-25T18:05:28.790 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 26 M 2024-03-25T18:05:28.790 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:05:29.551 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 61 kB/s | 31 kB 00:00 2024-03-25T18:05:29.584 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 28 kB/s | 15 kB 00:00 2024-03-25T18:05:29.726 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-25T18:05:29.760 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-25T18:05:30.119 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T18:05:30.186 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-03-25T18:05:30.361 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-03-25T18:05:30.428 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-03-25T18:05:30.495 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 461 kB/s | 31 kB 00:00 2024-03-25T18:05:30.587 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-25T18:05:31.504 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 486 kB/s | 863 kB 00:01 2024-03-25T18:05:36.737 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 237 kB/s | 1.8 MB 00:07 2024-03-25T18:05:36.743 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:36.743 INFO:teuthology.orchestra.run.smithi151.stdout:Total 883 kB/s | 6.9 MB 00:07 2024-03-25T18:05:36.763 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:05:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:05:36.782 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:05:36.961 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:05:36.961 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:05:37.491 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:05:37.579 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:05:37.615 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:05:37.653 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:05:37.684 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:05:37.784 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:05:37.919 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:05:38.013 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:05:38.065 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:05:38.094 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:05:38.117 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:05:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:05:38.393 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:05:39.190 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:05:39.190 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:05:39.190 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:05:39.190 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:05:39.190 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:05:39.190 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:05:39.190 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:05:39.190 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:05:39.191 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:05:39.191 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:05:39.191 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:05:39.191 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:05:39.550 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:05:39.550 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:39.550 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:05:39.550 INFO:teuthology.orchestra.run.smithi151.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:05:39.550 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:39.550 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:39.550 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:05:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:05:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:05:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:05:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:05:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:05:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:05:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:05:39.551 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:39.551 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:05:39.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:05:40.374 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:04 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:05:40.525 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:05:40.526 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-25T18:05:40.526 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-25T18:05:40.526 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-25T18:05:40.526 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:05:40.527 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:05:40.528 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:40.528 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:05:40.528 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-25T18:05:40.528 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2024-03-25T18:05:40.528 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:05:40.528 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 37 M 2024-03-25T18:05:40.529 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 209 M 2024-03-25T18:05:40.529 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:05:40.964 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 233 kB/s | 32 kB 00:00 2024-03-25T18:05:41.015 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 627 kB/s | 17 kB 00:00 2024-03-25T18:05:41.040 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T18:05:41.216 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T18:05:41.358 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T18:05:41.416 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-25T18:05:41.442 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T18:05:41.867 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-25T18:05:42.067 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-25T18:05:42.126 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 755 kB/s | 195 kB 00:00 2024-03-25T18:05:42.276 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-25T18:05:57.008 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 468 kB/s | 7.4 MB 00:16 2024-03-25T18:05:57.013 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:57.013 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.2 MB/s | 37 MB 00:16 2024-03-25T18:05:57.057 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:05:57.082 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:05:57.083 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:05:57.446 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:05:57.447 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:05:58.154 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:05:58.231 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:05:58.262 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:05:58.360 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:05:58.399 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:05:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:05:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:05:58.694 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:05:59.345 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:05:59.464 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:06:00.917 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:06:01.963 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:06:01.996 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:06:04.715 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:06:04.715 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:06:04.715 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:06:04.716 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:06:04.716 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:06:04.716 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:06:04.716 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:06:04.716 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:06:04.716 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:06:04.716 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:06:04.716 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:06:04.716 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:06:05.135 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:06:05.136 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:05.136 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:06:05.136 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:05.136 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:06:05.136 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:06:05.136 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:06:05.136 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:06:05.136 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:06:05.136 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:06:05.137 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:06:05.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:06:05.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:06:05.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:06:05.137 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:06:05.137 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:05.137 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:05.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:06:06.101 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:30 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:06.252 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:06.253 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-25T18:06:06.253 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-25T18:06:06.253 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-25T18:06:06.253 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:06:06.253 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:06:06.253 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T18:06:06.253 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:06:06.254 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-25T18:06:06.255 INFO:teuthology.orchestra.run.smithi151.stdout:Install 13 Packages 2024-03-25T18:06:06.255 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:06.255 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 2.0 M 2024-03-25T18:06:06.255 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 26 M 2024-03-25T18:06:06.255 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:06:06.584 INFO:teuthology.orchestra.run.smithi151.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 150 kB/s | 24 kB 00:00 2024-03-25T18:06:06.609 INFO:teuthology.orchestra.run.smithi151.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 96 kB/s | 18 kB 00:00 2024-03-25T18:06:06.710 INFO:teuthology.orchestra.run.smithi151.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 174 kB 00:00 2024-03-25T18:06:06.735 INFO:teuthology.orchestra.run.smithi151.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-03-25T18:06:06.761 INFO:teuthology.orchestra.run.smithi151.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:06:06.786 INFO:teuthology.orchestra.run.smithi151.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 849 kB/s | 43 kB 00:00 2024-03-25T18:06:07.053 INFO:teuthology.orchestra.run.smithi151.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-25T18:06:07.078 INFO:teuthology.orchestra.run.smithi151.stdout:(8/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 80 kB/s | 52 kB 00:00 2024-03-25T18:06:07.104 INFO:teuthology.orchestra.run.smithi151.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 94 kB/s | 32 kB 00:00 2024-03-25T18:06:07.454 INFO:teuthology.orchestra.run.smithi151.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 505 kB/s | 203 kB 00:00 2024-03-25T18:06:07.496 INFO:teuthology.orchestra.run.smithi151.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 151 kB/s | 59 kB 00:00 2024-03-25T18:06:07.588 INFO:teuthology.orchestra.run.smithi151.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 672 kB/s | 90 kB 00:00 2024-03-25T18:06:07.680 INFO:teuthology.orchestra.run.smithi151.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-25T18:06:07.682 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:07.682 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-03-25T18:06:07.683 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:06:07.697 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:06:07.697 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:06:07.795 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:06:07.797 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:06:08.366 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:06:08.536 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:06:08.590 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:06:08.654 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:06:08.692 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:06:08.733 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:06:08.827 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:06:08.945 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:06:09.002 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:06:09.723 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:06:09.767 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:06:09.801 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:06:09.849 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:06:09.871 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:06:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:06:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:06:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:06:11.050 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:06:11.051 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:06:11.052 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:06:11.052 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:06:11.052 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:11.052 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:11.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:06:11.686 INFO:teuthology.orchestra.run.smithi123.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 303 kB/s | 17 MB 00:56 2024-03-25T18:06:11.690 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:11.690 INFO:teuthology.orchestra.run.smithi123.stdout:Total 476 kB/s | 30 MB 01:03 2024-03-25T18:06:11.713 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:06:11.755 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:06:11.755 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:06:11.855 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:12.008 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:12.009 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-25T18:06:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-03-25T18:06:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:12.011 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 6.8 M 2024-03-25T18:06:12.011 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 33 M 2024-03-25T18:06:12.011 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:06:12.153 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:06:12.154 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:06:12.513 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T18:06:12.553 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 711 kB/s | 248 kB 00:00 2024-03-25T18:06:13.147 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 95 kB/s | 58 kB 00:00 2024-03-25T18:06:13.181 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 139 kB/s | 138 kB 00:00 2024-03-25T18:06:13.248 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 508 kB/s | 51 kB 00:00 2024-03-25T18:06:13.518 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:06:13.572 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:06:13.613 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:06:13.641 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:06:13.681 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 469 kB/s | 548 kB 00:01 2024-03-25T18:06:13.684 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:13.684 INFO:teuthology.orchestra.run.smithi151.stdout:Total 4.1 MB/s | 6.8 MB 00:01 2024-03-25T18:06:13.697 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:06:13.707 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:06:13.707 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:06:13.710 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:06:13.748 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:06:13.780 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:06:13.796 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:06:13.797 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:06:13.872 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:06:13.950 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:06:13.991 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:06:14.063 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:06:14.098 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:06:14.117 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:06:14.155 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:06:14.203 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:06:14.259 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:06:14.273 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:06:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:06:14.442 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:06:14.501 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:06:14.539 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:06:14.574 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:06:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:06:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:06:14.715 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:06:14.748 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:06:14.788 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:06:14.823 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:06:14.846 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:06:14.858 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:06:14.937 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:06:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:06:14.980 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:06:15.007 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:06:15.039 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:06:15.051 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:06:15.188 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:06:15.304 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:06:15.464 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:06:15.493 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:06:15.517 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:06:15.518 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:15.518 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:06:15.518 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:06:15.518 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:06:15.518 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:15.927 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:06:15.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:06:15.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:06:15.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:06:15.927 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:06:15.928 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:16.231 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:16.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse 2024-03-25T18:06:17.034 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:17.145 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:06:17.171 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:06:17.171 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:17.171 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:06:17.171 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:06:17.171 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:06:17.171 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:17.193 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:17.194 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:17.194 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-25T18:06:17.194 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:17.194 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:06:17.194 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:06:17.195 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T18:06:17.195 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:06:17.195 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:17.195 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:06:17.195 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:17.195 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-03-25T18:06:17.195 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:17.195 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 964 k 2024-03-25T18:06:17.196 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.8 M 2024-03-25T18:06:17.196 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:06:17.336 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 561 kB/s | 79 kB 00:00 2024-03-25T18:06:17.618 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:06:17.640 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:06:17.641 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:17.641 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:06:17.641 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:06:17.641 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:06:17.641 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:17.852 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:06:17.874 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:06:17.874 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:17.874 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:06:17.874 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:06:17.874 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:06:17.874 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:17.940 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:06:19.053 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 477 kB/s | 885 kB 00:01 2024-03-25T18:06:19.054 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:19.054 INFO:teuthology.orchestra.run.smithi151.stdout:Total 518 kB/s | 964 kB 00:01 2024-03-25T18:06:19.055 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:06:19.079 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:06:19.079 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:06:19.155 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:06:19.156 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:06:19.336 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:06:19.489 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:06:19.489 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:06:19.489 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:06:19.489 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:06:19.489 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:06:19.489 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:06:19.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:06:19.491 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:06:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:06:19.492 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:06:19.493 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:06:19.493 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:06:19.493 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:06:19.493 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:06:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:06:20.226 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:06:20.226 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:06:20.525 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:06:20.525 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:20.525 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:06:20.525 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:06:20.526 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:20.526 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:06:20.620 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:06:20.621 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:20.622 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:06:20.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-volume 2024-03-25T18:06:20.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-base 2024-03-25T18:06:21.366 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:45 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:21.525 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:21.526 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-25T18:06:21.526 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:21.526 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:06:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:06:21.527 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T18:06:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:06:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:21.527 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:06:21.527 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:21.527 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-03-25T18:06:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:21.527 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 332 k 2024-03-25T18:06:21.528 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.5 M 2024-03-25T18:06:21.528 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:06:21.539 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:05:47 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:06:21.636 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:21.645 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 589 kB/s | 69 kB 00:00 2024-03-25T18:06:21.688 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:06:21.689 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-25T18:06:21.689 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:06:21.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install cephadm 2024-03-25T18:06:22.313 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:05:48 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:06:22.461 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:06:22.462 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:06:22.462 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-25T18:06:22.462 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:06:22.462 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:06:22.462 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:06:22.462 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:22.463 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:06:22.463 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:06:22.463 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:06:22.463 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:22.463 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 747 k 2024-03-25T18:06:22.463 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 754 k 2024-03-25T18:06:22.463 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:06:22.645 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 236 kB/s | 264 kB 00:01 2024-03-25T18:06:22.647 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:22.647 INFO:teuthology.orchestra.run.smithi151.stdout:Total 297 kB/s | 332 kB 00:01 2024-03-25T18:06:22.647 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:06:22.657 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:06:22.657 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:06:22.699 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:06:22.699 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:06:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:06:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:06:23.082 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:06:23.097 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:06:23.097 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:23.098 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:06:23.098 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:23.856 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:06:24.058 INFO:teuthology.orchestra.run.smithi123.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 469 kB/s | 747 kB 00:01 2024-03-25T18:06:24.059 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:24.059 INFO:teuthology.orchestra.run.smithi123.stdout:Total 468 kB/s | 747 kB 00:01 2024-03-25T18:06:24.060 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:06:24.066 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:06:24.066 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:06:24.072 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:06:24.072 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:06:24.150 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:06:24.184 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:06:24.184 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:24.184 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:06:24.184 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:24.184 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:06:24.184 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:24.184 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:24.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel 2024-03-25T18:06:24.636 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:06:24.714 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:06:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:49 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:06:25.264 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 127 k 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 456 k 2024-03-25T18:06:25.265 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:06:25.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:06:25.446 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:25.446 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:06:25.446 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:25.446 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:25.446 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:06:25.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:06:25.980 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 177 kB/s | 127 kB 00:00 2024-03-25T18:06:25.981 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:25.981 INFO:teuthology.orchestra.run.smithi151.stdout:Total 177 kB/s | 127 kB 00:00 2024-03-25T18:06:25.981 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:06:25.988 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:06:25.988 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:06:26.046 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:06:26.046 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:06:26.194 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:06:26.218 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:05:52 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:06:26.313 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:26.372 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:06:26.373 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:06:26.373 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:06:26.373 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:06:26.373 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:06:26.373 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:06:26.373 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:26.373 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:06:26.373 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:06:26.373 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:06:26.373 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:26.374 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 145 k 2024-03-25T18:06:26.374 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 436 k 2024-03-25T18:06:26.374 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:06:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:27.047 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:06:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:27.047 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:27.047 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:27.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2 2024-03-25T18:06:27.880 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:51 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:27.980 INFO:teuthology.orchestra.run.smithi151.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:28.035 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:28.036 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-25T18:06:28.036 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:28.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs-devel 2024-03-25T18:06:28.665 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:52 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:28.815 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:28.815 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:28.815 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:06:28.815 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:28.815 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:06:28.815 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:06:28.815 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:28.816 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:06:28.816 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:28.816 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:06:28.816 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:28.816 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 31 k 2024-03-25T18:06:28.816 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 140 k 2024-03-25T18:06:28.816 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:06:29.363 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 57 kB/s | 31 kB 00:00 2024-03-25T18:06:29.363 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:29.364 INFO:teuthology.orchestra.run.smithi151.stdout:Total 57 kB/s | 31 kB 00:00 2024-03-25T18:06:29.364 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:06:29.367 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:06:29.367 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:06:29.397 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:06:29.398 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:06:29.509 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:06:29.611 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:30.025 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:30.286 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:30.286 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:30.286 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:06:30.286 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:30.286 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:30.286 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:30.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2 2024-03-25T18:06:31.034 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:31.131 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:31.182 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:31.183 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-25T18:06:31.183 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:31.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1 2024-03-25T18:06:31.793 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:31.891 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:31.943 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:31.944 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-25T18:06:31.944 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:32.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados 2024-03-25T18:06:32.569 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:32.666 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:32.720 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:32.721 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-25T18:06:32.721 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:32.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw 2024-03-25T18:06:33.383 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:33.482 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:33.534 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:33.536 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-25T18:06:33.536 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:33.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs 2024-03-25T18:06:33.596 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 20 kB/s | 145 kB 00:07 2024-03-25T18:06:33.597 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:33.598 INFO:teuthology.orchestra.run.smithi123.stdout:Total 20 kB/s | 145 kB 00:07 2024-03-25T18:06:33.598 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:06:33.607 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:06:33.607 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:06:33.663 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:06:33.663 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:06:33.808 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:06:33.864 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:06:33.888 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:06:33.888 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:33.889 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:06:33.889 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:34.161 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:58 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:34.259 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:34.311 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:34.312 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-25T18:06:34.312 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:34.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd 2024-03-25T18:06:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:06:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:34.845 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:06:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:34.845 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:06:34.919 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:58 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:35.017 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:35.068 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:35.069 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-25T18:06:35.069 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:35.073 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr 2024-03-25T18:06:35.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse 2024-03-25T18:06:35.645 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:06:01 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:06:35.684 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:59 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:35.741 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:35.793 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:06:35.795 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-25T18:06:35.795 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:06:35.831 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:35.832 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:35.832 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-25T18:06:35.832 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:35.832 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:06:35.832 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:06:35.832 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:35.832 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:06:35.832 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:35.833 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:06:35.833 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:35.833 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 85 k 2024-03-25T18:06:35.833 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 231 k 2024-03-25T18:06:35.833 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:06:35.861 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:06:36.415 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:06:02 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:06:36.563 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:06:36.564 INFO:teuthology.orchestra.run.smithi123.stdout:====================================================================================== 2024-03-25T18:06:36.564 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout:====================================================================================== 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:06:36.565 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:06:36.566 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:06:36.566 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:06:36.566 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:06:36.566 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:36.566 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:06:36.566 INFO:teuthology.orchestra.run.smithi123.stdout:====================================================================================== 2024-03-25T18:06:36.566 INFO:teuthology.orchestra.run.smithi123.stdout:Install 12 Packages 2024-03-25T18:06:36.566 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:36.567 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 6.9 M 2024-03-25T18:06:36.567 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 26 M 2024-03-25T18:06:36.567 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:06:36.742 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 94 kB/s | 85 kB 00:00 2024-03-25T18:06:36.742 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:36.743 INFO:teuthology.orchestra.run.smithi151.stdout:Total 94 kB/s | 85 kB 00:00 2024-03-25T18:06:36.743 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:06:36.751 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:06:36.751 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:06:36.809 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:06:36.810 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:06:36.946 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:06:37.048 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:37.316 INFO:teuthology.orchestra.run.smithi123.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 52 kB/s | 31 kB 00:00 2024-03-25T18:06:37.458 INFO:teuthology.orchestra.run.smithi123.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 20 kB/s | 15 kB 00:00 2024-03-25T18:06:37.488 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:37.492 INFO:teuthology.orchestra.run.smithi123.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-25T18:06:37.517 INFO:teuthology.orchestra.run.smithi123.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 4.0 MB/s | 238 kB 00:00 2024-03-25T18:06:37.792 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:37.792 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:37.792 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:06:37.792 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:37.792 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:37.792 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:38.009 INFO:teuthology.orchestra.run.smithi123.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T18:06:38.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror 2024-03-25T18:06:38.084 INFO:teuthology.orchestra.run.smithi123.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-25T18:06:38.293 INFO:teuthology.orchestra.run.smithi123.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.8 MB/s | 2.0 MB 00:00 2024-03-25T18:06:38.318 INFO:teuthology.orchestra.run.smithi123.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.0 MB/s | 863 kB 00:00 2024-03-25T18:06:38.368 INFO:teuthology.orchestra.run.smithi123.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.9 MB/s | 144 kB 00:00 2024-03-25T18:06:38.477 INFO:teuthology.orchestra.run.smithi123.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-25T18:06:38.585 INFO:teuthology.orchestra.run.smithi123.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 115 kB/s | 31 kB 00:00 2024-03-25T18:06:38.618 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:06:02 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:38.772 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:38.773 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:38.773 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:06:38.773 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:38.773 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:06:38.773 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:06:38.774 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:38.774 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:06:38.774 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:38.774 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:06:38.774 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:38.774 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 3.1 M 2024-03-25T18:06:38.774 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 13 M 2024-03-25T18:06:38.774 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:06:46.751 INFO:teuthology.orchestra.run.smithi123.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 182 kB/s | 1.8 MB 00:10 2024-03-25T18:06:46.756 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:46.756 INFO:teuthology.orchestra.run.smithi123.stdout:Total 689 kB/s | 6.9 MB 00:10 2024-03-25T18:06:46.776 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:06:46.796 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:06:46.796 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:06:46.978 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:06:46.979 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:06:47.493 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:06:47.560 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:06:47.595 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:06:47.641 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:06:47.680 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:06:47.786 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:06:47.913 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:06:48.000 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:06:48.043 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:06:48.080 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:06:48.111 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:06:48.352 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:06:48.378 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:06:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:06:49.166 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:06:49.544 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:06:49.544 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:49.544 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:06:49.544 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:06:49.544 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:49.544 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:49.544 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:49.545 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:06:49.545 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:06:49.545 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:06:49.545 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:06:49.545 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:06:49.545 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:06:49.545 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:06:49.545 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:06:49.545 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:49.545 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:06:49.814 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:06:50.374 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:06:16 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:06:50.523 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 269 kB/s | 3.1 MB 00:11 2024-03-25T18:06:50.524 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:50.524 INFO:teuthology.orchestra.run.smithi151.stdout:Total 269 kB/s | 3.1 MB 00:11 2024-03-25T18:06:50.525 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:06:50.525 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:06:50.525 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-03-25T18:06:50.525 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:06:50.526 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:06:50.527 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:06:50.527 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:06:50.527 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:06:50.527 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:06:50.527 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:50.527 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:06:50.527 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-03-25T18:06:50.527 INFO:teuthology.orchestra.run.smithi123.stdout:Install 12 Packages 2024-03-25T18:06:50.527 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:06:50.528 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 37 M 2024-03-25T18:06:50.528 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 209 M 2024-03-25T18:06:50.528 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:06:50.536 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:06:50.536 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:06:50.613 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:06:50.613 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:06:50.800 INFO:teuthology.orchestra.run.smithi123.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 283 kB/s | 32 kB 00:00 2024-03-25T18:06:50.858 INFO:teuthology.orchestra.run.smithi123.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 487 kB/s | 17 kB 00:00 2024-03-25T18:06:50.892 INFO:teuthology.orchestra.run.smithi123.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:06:51.043 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:06:51.067 INFO:teuthology.orchestra.run.smithi123.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T18:06:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:51.124 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:51.124 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:06:51.124 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:06:51.124 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:06:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:51.217 INFO:teuthology.orchestra.run.smithi123.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-25T18:06:51.259 INFO:teuthology.orchestra.run.smithi123.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T18:06:51.334 INFO:teuthology.orchestra.run.smithi123.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-03-25T18:06:51.660 INFO:teuthology.orchestra.run.smithi123.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-25T18:06:51.843 INFO:teuthology.orchestra.run.smithi123.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-25T18:06:51.902 INFO:teuthology.orchestra.run.smithi123.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-25T18:06:51.952 INFO:teuthology.orchestra.run.smithi123.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-25T18:06:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:52.268 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:06:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:52.268 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:52.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd 2024-03-25T18:06:53.111 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:53.260 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:53.260 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-25T18:06:53.260 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:53.260 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:06:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:06:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:53.261 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:06:53.261 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:53.261 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:06:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:53.262 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 171 k 2024-03-25T18:06:53.262 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 494 k 2024-03-25T18:06:53.262 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:06:54.263 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 171 kB/s | 171 kB 00:01 2024-03-25T18:06:54.264 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:54.264 INFO:teuthology.orchestra.run.smithi151.stdout:Total 171 kB/s | 171 kB 00:01 2024-03-25T18:06:54.264 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:06:54.273 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:06:54.273 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:06:54.331 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:06:54.331 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:06:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:06:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:55.329 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:06:55.329 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:55.330 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:55.330 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:06:55.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:06:56.153 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:06:20 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:06:56.306 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:06:56.307 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-03-25T18:06:56.308 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:06:56.308 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 91 M 2024-03-25T18:06:56.308 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 498 M 2024-03-25T18:06:56.308 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:07:20.773 INFO:teuthology.orchestra.run.smithi123.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 252 kB/s | 7.4 MB 00:30 2024-03-25T18:07:20.775 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:20.775 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.2 MB/s | 37 MB 00:30 2024-03-25T18:07:20.819 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:07:20.848 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:07:20.849 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:07:21.208 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:07:21.209 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:07:21.956 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:07:22.030 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:07:22.062 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:07:22.162 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:07:22.206 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:07:22.383 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:07:22.413 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:07:22.495 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:07:23.239 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:07:23.367 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:07:24.620 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:07:25.651 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:07:25.687 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:07:28.238 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:07:28.238 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:07:28.238 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:07:28.238 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:07:28.238 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:07:28.238 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:07:28.238 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:07:28.239 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:07:28.239 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:07:28.239 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:07:28.239 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:07:28.239 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:07:28.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:07:28.686 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:07:28.687 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:07:28.687 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:07:28.687 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:28.687 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:29.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:07:29.618 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:06:55 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:07:29.768 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:07:29.769 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================== 2024-03-25T18:07:29.769 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================== 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:07:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:07:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:07:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:07:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:07:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:07:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:29.771 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:07:29.771 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================== 2024-03-25T18:07:29.771 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2024-03-25T18:07:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:29.772 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 2.0 M 2024-03-25T18:07:29.772 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 26 M 2024-03-25T18:07:29.772 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:07:30.020 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 251 kB/s | 24 kB 00:00 2024-03-25T18:07:30.070 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-25T18:07:30.120 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T18:07:30.145 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:07:30.170 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:07:30.196 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 871 kB/s | 43 kB 00:00 2024-03-25T18:07:30.313 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-25T18:07:30.338 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-25T18:07:30.530 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 86 kB/s | 52 kB 00:00 2024-03-25T18:07:30.555 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 837 kB/s | 203 kB 00:00 2024-03-25T18:07:30.630 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-25T18:07:30.672 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-25T18:07:30.697 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 353 kB/s | 59 kB 00:00 2024-03-25T18:07:30.699 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:30.699 INFO:teuthology.orchestra.run.smithi123.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-25T18:07:30.700 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:07:30.713 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:07:30.713 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:07:30.802 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:07:30.802 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:07:31.315 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:07:31.479 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:07:31.540 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:07:31.595 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:07:31.624 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:07:31.674 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:07:31.767 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:07:31.867 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:07:31.925 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:07:32.649 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:07:32.697 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:07:32.730 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:07:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:07:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:07:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:07:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:07:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:07:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:07:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:07:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:07:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:07:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:07:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:07:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:07:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:07:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:07:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:07:33.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:07:33.946 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:07:33.946 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:07:33.946 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:07:33.946 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:33.946 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:34.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:07:34.760 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:00 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:07:34.910 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:07:34.910 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================== 2024-03-25T18:07:34.910 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-25T18:07:34.910 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================== 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================== 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2024-03-25T18:07:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:34.912 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 6.8 M 2024-03-25T18:07:34.912 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 33 M 2024-03-25T18:07:34.912 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:07:35.107 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.2 MB/s | 248 kB 00:00 2024-03-25T18:07:35.415 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-03-25T18:07:35.482 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 870 kB/s | 58 kB 00:00 2024-03-25T18:07:35.557 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 681 kB/s | 51 kB 00:00 2024-03-25T18:07:35.774 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 177 kB/s | 138 kB 00:00 2024-03-25T18:07:35.866 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.6 MB/s | 5.8 MB 00:00 2024-03-25T18:07:35.868 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:35.868 INFO:teuthology.orchestra.run.smithi123.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2024-03-25T18:07:35.881 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:07:35.891 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:07:35.891 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:07:35.979 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:07:35.981 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:07:36.272 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:07:36.439 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:07:36.494 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:07:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:07:37.089 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:07:37.158 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:07:37.182 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:07:37.965 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:07:37.966 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:07:37.966 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:07:37.966 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:07:37.966 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:07:37.966 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:07:38.257 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:07:38.257 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:38.258 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:07:38.258 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:07:38.258 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:07:38.258 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:07:38.258 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:07:38.258 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:07:38.258 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:07:38.258 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:38.258 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:38.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-fuse 2024-03-25T18:07:39.068 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:05 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:07:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:07:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:39.217 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-25T18:07:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:07:39.217 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:07:39.217 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T18:07:39.217 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:07:39.218 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:07:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:Install 2 Packages 2024-03-25T18:07:39.218 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 964 k 2024-03-25T18:07:39.218 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 2.8 M 2024-03-25T18:07:39.219 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:07:39.369 INFO:teuthology.orchestra.run.smithi123.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 521 kB/s | 79 kB 00:00 2024-03-25T18:07:44.185 INFO:teuthology.orchestra.run.smithi123.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 178 kB/s | 885 kB 00:04 2024-03-25T18:07:44.187 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:44.188 INFO:teuthology.orchestra.run.smithi123.stdout:Total 194 kB/s | 964 kB 00:04 2024-03-25T18:07:44.188 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:07:44.288 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:07:44.288 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:07:44.469 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:07:44.642 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:07:44.718 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:07:45.380 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:07:45.380 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:07:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:07:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:45.676 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:07:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:07:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:45.676 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:45.953 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-volume 2024-03-25T18:07:46.529 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:12 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:07:46.679 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:07:46.679 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:46.679 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout:Install 2 Packages 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 332 k 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 1.5 M 2024-03-25T18:07:46.680 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:07:46.790 INFO:teuthology.orchestra.run.smithi123.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 630 kB/s | 69 kB 00:00 2024-03-25T18:07:48.423 INFO:teuthology.orchestra.run.smithi123.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 151 kB/s | 264 kB 00:01 2024-03-25T18:07:48.425 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:48.425 INFO:teuthology.orchestra.run.smithi123.stdout:Total 190 kB/s | 332 kB 00:01 2024-03-25T18:07:48.425 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:07:48.435 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:07:48.435 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:07:48.478 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:07:48.479 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:07:48.648 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:07:48.830 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:07:48.867 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:07:48.882 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:07:48.882 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:07:48.882 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:07:48.882 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:49.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:07:49.909 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:07:49.909 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:49.909 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:07:49.909 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:07:49.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:07:49.909 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:49.909 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:50.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados-devel 2024-03-25T18:07:50.797 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:16 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:07:50.947 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:07:50.947 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:50.947 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:07:50.947 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:50.947 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:07:50.947 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:07:50.947 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:50.948 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:07:50.948 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:50.948 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:07:50.948 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:50.948 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 127 k 2024-03-25T18:07:50.948 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 456 k 2024-03-25T18:07:50.949 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:07:52.686 INFO:teuthology.orchestra.run.smithi123.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 73 kB/s | 127 kB 00:01 2024-03-25T18:07:52.687 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:52.687 INFO:teuthology.orchestra.run.smithi123.stdout:Total 73 kB/s | 127 kB 00:01 2024-03-25T18:07:52.688 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:07:52.694 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:07:52.694 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:07:52.754 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:07:52.754 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:07:52.901 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:07:53.002 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:53.446 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:53.722 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:53.722 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:53.722 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:07:53.722 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:53.722 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:53.722 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:53.953 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs2 2024-03-25T18:07:54.523 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:20 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:07:54.622 INFO:teuthology.orchestra.run.smithi123.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:07:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-25T18:07:54.679 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:54.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs-devel 2024-03-25T18:07:55.301 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:21 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:07:55.449 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:07:55.449 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 31 k 2024-03-25T18:07:55.450 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 140 k 2024-03-25T18:07:55.451 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:07:56.072 INFO:teuthology.orchestra.run.smithi123.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 50 kB/s | 31 kB 00:00 2024-03-25T18:07:56.073 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:56.073 INFO:teuthology.orchestra.run.smithi123.stdout:Total 50 kB/s | 31 kB 00:00 2024-03-25T18:07:56.073 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:07:56.077 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:07:56.077 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:07:56.111 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:07:56.111 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:07:56.238 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:07:56.354 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:56.754 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:56.996 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:56.996 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:56.996 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:07:56.996 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:56.996 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:07:56.996 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:57.169 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados2 2024-03-25T18:07:57.733 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:23 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:07:57.829 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:57.882 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:07:57.883 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-25T18:07:57.883 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:57.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librbd1 2024-03-25T18:07:58.531 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:24 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:07:58.633 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:58.686 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:07:58.688 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-25T18:07:58.688 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:58.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rados 2024-03-25T18:07:59.317 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:25 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:07:59.413 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:59.464 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:07:59.465 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-25T18:07:59.466 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:07:59.513 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rgw 2024-03-25T18:08:00.070 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:26 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:08:00.165 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:08:00.217 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:08:00.218 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-25T18:08:00.218 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:08:00.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-cephfs 2024-03-25T18:08:00.828 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:26 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:08:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:08:00.975 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:08:00.976 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-25T18:08:00.976 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:08:01.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rbd 2024-03-25T18:08:01.581 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:27 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:08:01.678 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:08:01.729 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:08:01.730 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-25T18:08:01.730 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:08:01.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-fuse 2024-03-25T18:08:02.341 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:28 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:08:02.488 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 85 k 2024-03-25T18:08:02.489 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 231 k 2024-03-25T18:08:02.490 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:08:03.442 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 90 kB/s | 85 kB 00:00 2024-03-25T18:08:03.442 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:03.443 INFO:teuthology.orchestra.run.smithi123.stdout:Total 89 kB/s | 85 kB 00:00 2024-03-25T18:08:03.443 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:08:03.450 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:08:03.451 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:08:03.508 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:08:03.508 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:08:03.645 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:08:03.756 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:04.159 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:04.462 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:04.463 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:04.463 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:08:04.463 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:04.463 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:04.463 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:08:04.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-mirror 2024-03-25T18:08:05.241 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:31 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:08:05.390 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:08:05.390 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:05.390 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 3.1 M 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 13 M 2024-03-25T18:08:05.391 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:08:10.669 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 599 kB/s | 3.1 MB 00:05 2024-03-25T18:08:10.669 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:10.670 INFO:teuthology.orchestra.run.smithi123.stdout:Total 599 kB/s | 3.1 MB 00:05 2024-03-25T18:08:10.670 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:08:10.681 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:08:10.681 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:08:10.758 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:08:10.758 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:08:11.027 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 308 kB/s | 22 MB 01:14 2024-03-25T18:08:11.200 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:08:11.372 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:11.396 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:11.397 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:08:11.397 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:08:11.397 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:08:11.397 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:08:11.397 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:12.420 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:08:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:12.421 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:08:12.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-nbd 2024-03-25T18:08:13.247 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:39 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:08:13.396 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 171 k 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 494 k 2024-03-25T18:08:13.397 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:08:14.173 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 222 kB/s | 171 kB 00:00 2024-03-25T18:08:14.173 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:14.173 INFO:teuthology.orchestra.run.smithi123.stdout:Total 221 kB/s | 171 kB 00:00 2024-03-25T18:08:14.174 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:08:14.181 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:08:14.181 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:08:14.238 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:08:14.239 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:08:14.388 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:08:14.503 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:14.931 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:15.212 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:15.212 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:15.212 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:08:15.212 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:15.212 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:15.213 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:08:15.439 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:08:16.013 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:07:42 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:08:16.166 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:08:16.166 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout:Install 2 Packages 2024-03-25T18:08:16.167 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:08:16.168 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 91 M 2024-03-25T18:08:16.168 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 498 M 2024-03-25T18:08:16.168 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:09:16.853 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 496 kB/s | 68 MB 02:20 2024-03-25T18:09:16.854 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:16.854 INFO:teuthology.orchestra.run.smithi151.stdout:Total 660 kB/s | 91 MB 02:20 2024-03-25T18:09:16.854 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:09:16.858 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:09:16.859 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:09:17.442 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:09:17.443 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:09:20.550 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:09:25.657 INFO:teuthology.orchestra.run.smithi123.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 331 kB/s | 22 MB 01:09 2024-03-25T18:09:27.683 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:09:27.763 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:09:31.376 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:09:31.376 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:09:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:09:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:09:31.637 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:09:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:09:31.638 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:09:31.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:09:32.442 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:56 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:09:32.590 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:09:32.591 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:09:32.591 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:09:32.591 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:09:32.591 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:09:32.591 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:09:32.591 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:09:32.591 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:09:32.591 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:09:32.591 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:09:32.592 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:09:32.592 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 416 M 2024-03-25T18:09:32.592 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.7 G 2024-03-25T18:09:32.592 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:11:34.207 INFO:teuthology.orchestra.run.smithi123.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 352 kB/s | 68 MB 03:18 2024-03-25T18:11:34.208 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:34.208 INFO:teuthology.orchestra.run.smithi123.stdout:Total 468 kB/s | 91 MB 03:18 2024-03-25T18:11:34.208 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:11:34.213 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:11:34.213 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:11:34.804 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:11:34.804 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:11:37.924 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:11:45.084 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:11:45.232 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:11:48.472 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:11:48.472 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:11:48.734 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:11:48.734 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:11:48.734 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:11:48.734 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:48.734 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:48.734 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:11:48.734 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:11:48.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:11:49.507 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:11:15 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:11:49.655 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:11:49.656 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:11:49.656 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:11:49.656 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:11:49.656 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:11:49.656 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:11:49.656 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:11:49.656 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:11:49.656 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:11:49.657 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:11:49.657 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:11:49.657 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 416 M 2024-03-25T18:11:49.657 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 1.7 G 2024-03-25T18:11:49.657 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:21:06.965 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 614 kB/s | 416 MB 11:34 2024-03-25T18:21:06.966 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:06.966 INFO:teuthology.orchestra.run.smithi151.stdout:Total 614 kB/s | 416 MB 11:34 2024-03-25T18:21:06.967 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:21:06.970 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:21:06.970 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:21:09.445 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:21:09.445 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:21:51.612 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:21:51.710 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:00.212 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:00.479 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:00.479 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:22:00.479 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:22:00.479 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:00.479 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:22:00.479 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:22:00.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:22:01.309 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:21:25 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:22:01.466 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:22:01.466 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 1.4 M 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 6.4 M 2024-03-25T18:22:01.467 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:22:05.040 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 409 kB/s | 1.4 MB 00:03 2024-03-25T18:22:05.041 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:05.041 INFO:teuthology.orchestra.run.smithi151.stdout:Total 409 kB/s | 1.4 MB 00:03 2024-03-25T18:22:05.041 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:22:05.046 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:22:05.046 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:22:05.091 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:22:05.092 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:22:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:22:05.463 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:22:05.878 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:22:06.154 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:22:06.154 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:22:06.154 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:22:06.154 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:06.154 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:22:06.154 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:22:06.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:22:06.910 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:21:30 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:22:07.059 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:22:07.059 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:22:07.059 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 176 M 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 753 M 2024-03-25T18:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:23:02.359 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 633 kB/s | 416 MB 11:12 2024-03-25T18:23:02.359 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:02.359 INFO:teuthology.orchestra.run.smithi123.stdout:Total 633 kB/s | 416 MB 11:12 2024-03-25T18:23:02.359 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:23:02.362 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:23:02.362 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:23:04.836 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:23:04.836 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:23:47.198 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:23:47.495 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:55.676 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:55.932 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:55.933 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:23:55.933 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:23:55.933 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:55.933 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:23:55.933 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:23:56.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:23:56.724 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:23:22 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:23:56.871 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:23:56.871 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================== 2024-03-25T18:23:56.871 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:23:56.871 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================== 2024-03-25T18:23:56.871 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:23:56.871 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:23:56.871 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:23:56.871 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:23:56.871 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================== 2024-03-25T18:23:56.872 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:23:56.872 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:23:56.872 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 1.4 M 2024-03-25T18:23:56.872 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 6.4 M 2024-03-25T18:23:56.872 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:24:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 450 kB/s | 1.4 MB 00:03 2024-03-25T18:24:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:Total 449 kB/s | 1.4 MB 00:03 2024-03-25T18:24:00.123 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:24:00.127 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:24:00.127 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:24:00.172 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:24:00.172 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:24:00.428 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:24:00.536 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:24:00.947 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:24:01.219 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:24:01.219 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:24:01.219 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:24:01.219 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:01.219 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:24:01.219 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:24:01.406 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:24:01.959 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:23:27 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:24:02.107 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:24:02.108 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:24:02.108 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:24:02.108 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:24:02.108 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:24:02.108 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:24:02.108 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:24:02.109 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:24:02.109 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:24:02.109 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:24:02.109 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:24:02.109 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 176 M 2024-03-25T18:24:02.109 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 753 M 2024-03-25T18:24:02.109 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:27:17.513 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 579 kB/s | 176 MB 05:10 2024-03-25T18:27:17.514 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:27:17.514 INFO:teuthology.orchestra.run.smithi151.stdout:Total 579 kB/s | 176 MB 05:10 2024-03-25T18:27:17.514 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:27:17.517 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:27:17.517 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:27:18.575 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:27:18.575 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:27:36.782 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:27:37.027 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:27:41.568 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 820 kB/s | 176 MB 03:39 2024-03-25T18:27:41.569 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:27:41.569 INFO:teuthology.orchestra.run.smithi123.stdout:Total 820 kB/s | 176 MB 03:39 2024-03-25T18:27:41.570 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:27:41.573 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:27:41.573 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:27:41.790 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:27:42.054 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:27:42.054 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:27:42.054 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:27:42.054 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:27:42.054 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:27:42.054 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:27:42.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:27:42.630 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:27:42.630 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:27:42.822 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:27:06 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:27:42.979 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:27:42.980 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 672 M 2024-03-25T18:27:42.981 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 3.1 G 2024-03-25T18:27:42.981 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:28:00.823 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:28:01.073 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:28:05.708 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:28:05.968 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:28:05.969 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:28:05.969 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:28:05.969 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:28:05.969 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:28:05.969 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:28:06.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:28:06.734 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:27:32 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:28:06.883 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:28:06.884 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:28:06.884 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:28:06.884 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:28:06.884 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:28:06.884 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:28:06.884 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:28:06.884 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:28:06.884 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:28:06.884 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:28:06.884 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:28:06.885 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 672 M 2024-03-25T18:28:06.885 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 3.1 G 2024-03-25T18:28:06.885 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:52:56.324 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 462 kB/s | 672 MB 24:49 2024-03-25T18:52:56.324 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:56.325 INFO:teuthology.orchestra.run.smithi123.stdout:Total 462 kB/s | 672 MB 24:49 2024-03-25T18:52:56.325 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:52:56.329 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:52:56.329 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:53:00.290 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:53:00.291 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:54:11.004 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:54:11.182 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:54:20.854 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:54:21.104 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:54:21.104 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:54:21.105 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:54:21.105 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:54:21.105 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:54:21.105 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:54:21.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:54:21.955 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:53:47 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:54:22.059 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:54:22.115 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:54:22.116 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-25T18:54:22.116 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:54:22.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:54:22.779 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:53:48 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:54:22.933 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:54:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:54:22.934 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:54:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:54:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:54:22.934 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:54:22.934 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:54:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:54:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:54:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:54:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:54:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 20 M 2024-03-25T18:54:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 103 M 2024-03-25T18:54:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:55:13.052 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 417 kB/s | 672 MB 27:30 2024-03-25T18:55:13.053 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:55:13.053 INFO:teuthology.orchestra.run.smithi151.stdout:Total 417 kB/s | 672 MB 27:30 2024-03-25T18:55:13.054 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:55:13.058 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:55:13.058 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:55:17.031 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:55:17.031 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:55:25.351 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 323 kB/s | 20 MB 01:02 2024-03-25T18:55:25.351 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:55:25.352 INFO:teuthology.orchestra.run.smithi123.stdout:Total 323 kB/s | 20 MB 01:02 2024-03-25T18:55:25.352 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:55:25.356 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:55:25.356 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:55:25.505 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:55:25.505 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:55:27.734 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:55:27.841 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:55:28.765 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:55:29.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:55:29.022 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:55:29.022 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:55:29.022 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:55:29.022 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:55:29.022 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:55:29.191 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:55:29.750 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:54:55 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:55:29.900 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 48 M 2024-03-25T18:55:29.901 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 173 M 2024-03-25T18:55:29.902 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:56:28.112 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:56:28.680 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:56:38.012 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:56:38.284 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:56:38.284 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:56:38.284 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:56:38.284 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:38.284 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:56:38.284 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:56:38.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:56:39.131 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:56:03 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:56:39.228 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:56:39.281 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:56:39.282 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-25T18:56:39.282 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:56:39.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:56:39.907 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:56:03 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:56:40.057 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:56:40.058 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:56:40.058 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:56:40.058 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:56:40.059 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:56:40.059 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:56:40.059 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:56:40.059 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:56:40.059 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:56:40.059 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:56:40.059 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:56:40.059 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 20 M 2024-03-25T18:56:40.059 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 103 M 2024-03-25T18:56:40.059 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:56:48.534 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 621 kB/s | 48 MB 01:18 2024-03-25T18:56:48.534 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:56:48.535 INFO:teuthology.orchestra.run.smithi123.stdout:Total 621 kB/s | 48 MB 01:18 2024-03-25T18:56:48.535 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T18:56:48.539 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T18:56:48.539 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T18:56:48.850 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T18:56:48.851 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T18:56:53.597 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T18:56:53.732 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:56:55.057 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:56:55.310 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:56:55.310 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:56:55.311 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T18:56:55.311 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:55.311 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:56:55.311 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T18:56:55.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:56:56.058 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:56:22 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T18:56:56.206 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T18:56:56.207 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:56:56.207 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T18:56:56.207 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:56:56.207 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T18:56:56.207 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:56:56.207 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:56:56.208 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T18:56:56.208 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T18:56:56.208 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T18:56:56.208 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T18:56:56.208 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 99 M 2024-03-25T18:56:56.208 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 449 M 2024-03-25T18:56:56.208 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T18:57:29.233 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 410 kB/s | 20 MB 00:49 2024-03-25T18:57:29.234 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:57:29.234 INFO:teuthology.orchestra.run.smithi151.stdout:Total 410 kB/s | 20 MB 00:49 2024-03-25T18:57:29.235 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:57:29.238 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:57:29.238 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:57:29.390 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:57:29.390 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:57:31.617 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:57:31.797 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:57:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:57:33.047 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:57:33.047 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:57:33.048 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:57:33.048 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:57:33.048 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:57:33.048 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:57:33.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:57:33.781 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:56:57 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:57:33.930 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:57:33.931 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:57:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:57:33.931 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:57:33.931 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:57:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:57:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:57:33.931 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:57:33.931 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:57:33.931 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:57:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:57:33.932 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 48 M 2024-03-25T18:57:33.932 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 173 M 2024-03-25T18:57:33.932 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T18:59:25.354 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 439 kB/s | 48 MB 01:51 2024-03-25T18:59:25.356 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:59:25.356 INFO:teuthology.orchestra.run.smithi151.stdout:Total 439 kB/s | 48 MB 01:51 2024-03-25T18:59:25.356 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T18:59:25.358 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T18:59:25.359 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T18:59:25.671 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T18:59:25.671 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T18:59:30.397 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T18:59:30.493 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:59:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:59:32.147 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:59:32.148 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:59:32.148 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T18:59:32.148 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:59:32.148 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:59:32.148 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T18:59:32.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:59:32.898 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:58:56 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T18:59:33.045 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T18:59:33.046 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:59:33.046 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T18:59:33.046 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:59:33.046 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T18:59:33.046 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:59:33.046 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:59:33.047 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T18:59:33.047 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T18:59:33.047 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T18:59:33.047 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T18:59:33.047 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 99 M 2024-03-25T18:59:33.047 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 449 M 2024-03-25T18:59:33.047 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:00:19.783 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 496 kB/s | 99 MB 03:23 2024-03-25T19:00:19.783 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:00:19.784 INFO:teuthology.orchestra.run.smithi123.stdout:Total 496 kB/s | 99 MB 03:23 2024-03-25T19:00:19.784 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:00:19.787 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:00:19.787 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:00:20.398 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:00:20.398 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:00:30.723 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:00:30.830 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:33.873 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:33.873 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:00:33.874 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:00:33.874 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:00:33.874 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:00:33.874 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:00:34.063 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:00:34.623 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:00:00 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:00:34.771 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:00:34.772 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:00:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:00:34.772 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:00:34.772 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:00:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:00:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:00:34.773 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:00:34.773 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:00:34.773 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:00:34.773 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:00:34.773 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 287 M 2024-03-25T19:00:34.773 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 1.2 G 2024-03-25T19:00:34.773 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:03:09.778 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 466 kB/s | 99 MB 03:36 2024-03-25T19:03:09.779 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:03:09.779 INFO:teuthology.orchestra.run.smithi151.stdout:Total 466 kB/s | 99 MB 03:36 2024-03-25T19:03:09.779 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:03:09.782 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:03:09.782 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:03:10.389 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:03:10.389 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:03:20.663 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:03:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:03:23.845 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:03:24.111 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:03:24.112 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:03:24.112 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:03:24.112 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:03:24.112 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:03:24.112 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:03:24.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:03:24.868 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:02:48 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:03:25.017 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:03:25.018 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:03:25.018 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:03:25.018 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:03:25.018 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:03:25.018 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:03:25.018 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:03:25.018 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:03:25.018 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:03:25.019 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:03:25.019 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:03:25.019 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 287 M 2024-03-25T19:03:25.019 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.2 G 2024-03-25T19:03:25.019 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:12:39.941 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 405 kB/s | 287 MB 12:05 2024-03-25T19:12:39.942 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:39.942 INFO:teuthology.orchestra.run.smithi123.stdout:Total 405 kB/s | 287 MB 12:05 2024-03-25T19:12:39.942 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:12:39.944 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:12:39.945 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:12:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:12:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:13:11.044 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:13:11.152 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:13:18.501 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:13:18.766 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:13:18.766 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:18.766 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:13:18.766 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:13:18.766 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:18.766 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:13:18.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:13:19.520 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:12:45 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:13:19.667 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:13:19.668 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:19.668 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:13:19.668 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:19.668 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:13:19.668 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:13:19.668 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:19.669 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:13:19.669 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:19.669 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:13:19.669 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:19.669 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 5.9 M 2024-03-25T19:13:19.669 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 28 M 2024-03-25T19:13:19.669 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:13:34.987 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 397 kB/s | 5.9 MB 00:15 2024-03-25T19:13:34.988 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:13:34.988 INFO:teuthology.orchestra.run.smithi123.stdout:Total 397 kB/s | 5.9 MB 00:15 2024-03-25T19:13:34.988 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:13:34.992 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:13:34.993 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:13:35.064 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:13:35.064 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:13:35.799 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:13:35.903 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:13:36.446 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:13:36.696 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:13:36.696 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:36.696 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:13:36.696 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:13:36.696 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:36.697 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:13:36.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:13:37.454 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:13:03 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:13:37.604 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 109 k 2024-03-25T19:13:37.605 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 440 k 2024-03-25T19:13:37.606 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:13:38.415 INFO:teuthology.orchestra.run.smithi123.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 134 kB/s | 109 kB 00:00 2024-03-25T19:13:38.415 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:13:38.416 INFO:teuthology.orchestra.run.smithi123.stdout:Total 134 kB/s | 109 kB 00:00 2024-03-25T19:13:38.416 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:13:38.419 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:13:38.419 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:13:38.455 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:13:38.456 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:13:38.575 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:13:38.680 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:13:39.045 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:13:39.304 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:13:39.305 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:39.305 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:13:39.305 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:13:39.305 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:39.305 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:13:39.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:13:40.040 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:13:06 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:13:40.189 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:13:40.190 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:40.190 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:13:40.190 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:40.190 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:13:40.190 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:13:40.190 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:40.190 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:13:40.191 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:40.191 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:13:40.191 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:40.191 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 5.4 M 2024-03-25T19:13:40.191 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 25 M 2024-03-25T19:13:40.192 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 567 kB/s | 5.4 MB 00:09 2024-03-25T19:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout:Total 567 kB/s | 5.4 MB 00:09 2024-03-25T19:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:13:49.923 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:13:49.924 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:13:50.586 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:13:50.690 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:51.218 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:51.476 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:51.476 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:51.476 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:13:51.477 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:13:51.477 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:51.477 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:13:51.667 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados2-debuginfo 2024-03-25T19:13:52.231 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:13:18 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:13:52.379 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:13:52.379 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 44 M 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 253 M 2024-03-25T19:13:52.380 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:14:27.047 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 444 kB/s | 287 MB 11:02 2024-03-25T19:14:27.048 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:14:27.048 INFO:teuthology.orchestra.run.smithi151.stdout:Total 444 kB/s | 287 MB 11:02 2024-03-25T19:14:27.048 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:14:27.052 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:14:27.052 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:14:28.757 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:14:28.758 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:14:58.056 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:14:58.167 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:15:06.090 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:15:06.364 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:15:06.364 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:06.364 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:15:06.364 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:15:06.364 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:06.365 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:15:06.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:15:07.132 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:14:31 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:15:07.281 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:15:07.281 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:07.281 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:15:07.281 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:07.281 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:15:07.282 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:15:07.282 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:07.282 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:15:07.282 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:07.282 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:15:07.282 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:07.282 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 5.9 M 2024-03-25T19:15:07.283 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 28 M 2024-03-25T19:15:07.283 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:15:11.550 INFO:teuthology.orchestra.run.smithi123.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 573 kB/s | 44 MB 01:19 2024-03-25T19:15:11.550 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:15:11.551 INFO:teuthology.orchestra.run.smithi123.stdout:Total 573 kB/s | 44 MB 01:19 2024-03-25T19:15:11.551 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:15:11.555 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:15:11.555 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:15:11.853 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:15:11.854 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:15:16.914 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:15:17.015 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:18.794 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:19.051 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:19.051 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:15:19.051 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:15:19.051 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:15:19.051 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:15:19.051 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:15:19.265 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:15:19.830 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:14:45 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:15:19.983 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:15:19.984 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:15:19.984 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:15:19.984 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:15:19.984 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:15:19.984 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:15:19.984 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:15:19.984 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:15:19.984 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:15:19.984 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:15:19.985 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:15:19.985 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 52 M 2024-03-25T19:15:19.985 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 308 M 2024-03-25T19:15:19.985 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:15:27.044 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 308 kB/s | 5.9 MB 00:19 2024-03-25T19:15:27.044 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:15:27.045 INFO:teuthology.orchestra.run.smithi151.stdout:Total 308 kB/s | 5.9 MB 00:19 2024-03-25T19:15:27.045 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:15:27.049 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:15:27.049 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:15:27.121 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:15:27.122 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:15:27.857 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:15:27.958 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:28.530 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:28.799 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:28.799 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:28.799 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:15:28.799 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:15:28.799 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:28.799 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:15:28.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:15:29.546 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:14:53 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:15:29.699 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:29.700 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 109 k 2024-03-25T19:15:29.701 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 440 k 2024-03-25T19:15:29.701 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:15:30.755 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 103 kB/s | 109 kB 00:01 2024-03-25T19:15:30.755 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:15:30.756 INFO:teuthology.orchestra.run.smithi151.stdout:Total 103 kB/s | 109 kB 00:01 2024-03-25T19:15:30.756 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:15:30.760 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:15:30.760 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:15:30.799 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:15:30.799 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:15:30.924 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:15:31.029 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:15:31.406 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:15:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:15:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:31.672 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:15:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:15:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:31.672 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:15:31.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:15:32.424 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:14:56 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:15:32.572 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:15:32.573 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:32.573 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:15:32.573 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:32.573 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:15:32.574 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:15:32.574 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:32.574 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:15:32.574 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:32.574 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:15:32.574 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:32.574 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 5.4 M 2024-03-25T19:15:32.574 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 25 M 2024-03-25T19:15:32.574 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:15:43.373 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 508 kB/s | 5.4 MB 00:10 2024-03-25T19:15:43.374 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:15:43.374 INFO:teuthology.orchestra.run.smithi151.stdout:Total 507 kB/s | 5.4 MB 00:10 2024-03-25T19:15:43.375 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:15:43.378 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:15:43.379 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:15:43.445 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:15:43.446 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:15:44.139 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:15:44.254 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:15:44.789 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:15:45.062 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:15:45.062 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:45.062 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:15:45.062 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:15:45.063 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:45.063 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:15:45.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2-debuginfo 2024-03-25T19:15:45.828 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:15:09 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:15:45.977 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:15:45.978 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:15:45.978 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:45.978 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:15:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:15:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:45.978 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:15:45.978 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:15:45.978 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:15:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:15:45.979 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 44 M 2024-03-25T19:15:45.979 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 253 M 2024-03-25T19:15:45.979 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:17:03.091 INFO:teuthology.orchestra.run.smithi151.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 588 kB/s | 44 MB 01:17 2024-03-25T19:17:03.092 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:03.092 INFO:teuthology.orchestra.run.smithi151.stdout:Total 588 kB/s | 44 MB 01:17 2024-03-25T19:17:03.092 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:17:03.096 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:17:03.096 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:17:03.395 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:17:03.395 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:17:08.522 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:17:10.601 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:17:10.882 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:17:10.883 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:17:10.883 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:17:10.883 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:10.883 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:17:10.883 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:17:11.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:17:11.663 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:16:35 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:17:11.812 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:17:11.813 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:17:11.813 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:17:11.813 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:17:11.813 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:17:11.813 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:17:11.813 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:17:11.813 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:17:11.813 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:17:11.814 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:17:11.814 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:17:11.814 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 52 M 2024-03-25T19:17:11.814 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 308 M 2024-03-25T19:17:11.814 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:17:37.625 INFO:teuthology.orchestra.run.smithi123.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 386 kB/s | 52 MB 02:17 2024-03-25T19:17:37.626 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:37.626 INFO:teuthology.orchestra.run.smithi123.stdout:Total 386 kB/s | 52 MB 02:17 2024-03-25T19:17:37.626 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:17:37.630 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:17:37.630 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:17:37.973 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:17:37.973 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:17:43.983 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:17:44.189 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:17:46.347 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:17:46.607 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:17:46.607 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:46.607 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:17:46.607 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:46.607 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:46.608 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:17:46.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:17:47.334 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:17:13 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:17:47.483 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:17:47.483 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 615 k 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 2.1 M 2024-03-25T19:17:47.484 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:17:51.169 INFO:teuthology.orchestra.run.smithi123.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 167 kB/s | 615 kB 00:03 2024-03-25T19:17:51.169 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:51.170 INFO:teuthology.orchestra.run.smithi123.stdout:Total 167 kB/s | 615 kB 00:03 2024-03-25T19:17:51.170 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:17:51.174 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:17:51.174 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:17:51.215 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:17:51.215 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:17:51.386 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:17:51.492 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:17:51.860 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:17:52.112 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:17:52.112 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:52.112 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:17:52.112 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:52.112 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:52.112 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:17:52.308 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:17:52.867 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:17:18 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:17:53.018 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:53.019 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 771 k 2024-03-25T19:17:53.020 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 2.7 M 2024-03-25T19:17:53.020 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:17:56.088 INFO:teuthology.orchestra.run.smithi123.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 251 kB/s | 771 kB 00:03 2024-03-25T19:17:56.088 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:56.089 INFO:teuthology.orchestra.run.smithi123.stdout:Total 251 kB/s | 771 kB 00:03 2024-03-25T19:17:56.089 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:17:56.093 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:17:56.093 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:17:56.134 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:17:56.135 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:17:56.317 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:17:56.420 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:17:56.809 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:17:57.064 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:17:57.064 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:17:57.064 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:57.064 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:57.064 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:17:57.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:17:57.831 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:17:23 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:17:57.984 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:17:57.984 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:17:57.984 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:17:57.984 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:17:57.984 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:17:57.984 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:17:57.985 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:57.985 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:17:57.985 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:17:57.985 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:17:57.985 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:17:57.985 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 1.1 M 2024-03-25T19:17:57.985 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 4.3 M 2024-03-25T19:17:57.986 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:18:00.340 INFO:teuthology.orchestra.run.smithi123.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 489 kB/s | 1.1 MB 00:02 2024-03-25T19:18:00.341 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:00.341 INFO:teuthology.orchestra.run.smithi123.stdout:Total 488 kB/s | 1.1 MB 00:02 2024-03-25T19:18:00.342 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:18:00.345 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:18:00.345 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:18:00.389 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:18:00.389 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:18:00.614 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:18:00.716 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:18:01.106 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:18:01.360 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:18:01.360 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:01.360 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:18:01.361 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:01.361 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:01.361 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:18:01.564 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:18:02.125 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:17:28 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:18:02.275 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:18:02.276 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:18:02.276 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:18:02.276 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:18:02.276 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:18:02.276 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:18:02.276 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:02.276 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:18:02.276 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:18:02.276 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:18:02.276 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:02.277 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 337 k 2024-03-25T19:18:02.277 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 1.0 M 2024-03-25T19:18:02.277 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:18:03.595 INFO:teuthology.orchestra.run.smithi123.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 256 kB/s | 337 kB 00:01 2024-03-25T19:18:03.596 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:03.596 INFO:teuthology.orchestra.run.smithi123.stdout:Total 256 kB/s | 337 kB 00:01 2024-03-25T19:18:03.596 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:18:03.600 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:18:03.600 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:18:03.640 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:18:03.641 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:18:03.781 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:18:03.878 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:18:04.259 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:18:04.518 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:18:04.518 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:04.518 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:18:04.518 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:04.518 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:04.518 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:18:04.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:18:05.284 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:17:31 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:18:05.434 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:18:05.435 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:18:05.435 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:18:05.435 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:18:05.435 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:18:05.435 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:18:05.435 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:05.436 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:18:05.436 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:18:05.436 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:18:05.436 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:05.436 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 509 k 2024-03-25T19:18:05.436 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 2.5 M 2024-03-25T19:18:05.436 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:18:06.440 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 507 kB/s | 509 kB 00:01 2024-03-25T19:18:06.441 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:06.441 INFO:teuthology.orchestra.run.smithi123.stdout:Total 506 kB/s | 509 kB 00:01 2024-03-25T19:18:06.441 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:18:06.444 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:18:06.445 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:18:06.483 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:18:06.483 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:18:06.653 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:18:06.772 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:07.160 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:07.418 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:07.418 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:07.418 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:18:07.418 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:07.418 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:07.418 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:18:07.595 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:18:08.154 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:17:34 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:18:08.302 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:18:08.302 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:18:08.302 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:18:08.302 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:18:08.303 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:18:08.303 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:18:08.303 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:08.303 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:18:08.303 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:18:08.303 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:18:08.303 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:18:08.304 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 52 M 2024-03-25T19:18:08.304 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 298 M 2024-03-25T19:18:08.304 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:19:22.943 INFO:teuthology.orchestra.run.smithi151.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 405 kB/s | 52 MB 02:11 2024-03-25T19:19:22.944 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:22.944 INFO:teuthology.orchestra.run.smithi151.stdout:Total 405 kB/s | 52 MB 02:11 2024-03-25T19:19:22.944 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:19:22.947 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:19:22.947 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:19:23.290 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:19:23.291 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:19:29.265 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:19:29.400 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:19:31.620 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:19:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:19:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:31.896 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:19:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:31.896 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:19:32.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:19:32.674 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:18:56 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:19:32.824 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:19:32.825 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:19:32.825 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:32.825 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:19:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:19:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:32.825 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:19:32.825 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:32.825 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:19:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:32.826 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 615 k 2024-03-25T19:19:32.826 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.1 M 2024-03-25T19:19:32.826 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:19:33.962 INFO:teuthology.orchestra.run.smithi151.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 541 kB/s | 615 kB 00:01 2024-03-25T19:19:33.962 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:33.963 INFO:teuthology.orchestra.run.smithi151.stdout:Total 541 kB/s | 615 kB 00:01 2024-03-25T19:19:33.963 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:19:33.967 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:19:33.967 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:19:34.007 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:19:34.007 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:19:34.176 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:19:34.269 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:19:34.707 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:19:35.004 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:19:35.004 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:35.004 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:19:35.004 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:35.004 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:35.004 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:19:35.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:19:35.779 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:18:59 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:19:35.930 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:19:35.930 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:35.931 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:19:35.931 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:35.931 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:19:35.931 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:19:35.931 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:35.931 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:19:35.931 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:35.931 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:19:35.931 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:35.932 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 771 k 2024-03-25T19:19:35.932 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.7 M 2024-03-25T19:19:35.932 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:19:37.140 INFO:teuthology.orchestra.run.smithi151.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 638 kB/s | 771 kB 00:01 2024-03-25T19:19:37.141 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:37.141 INFO:teuthology.orchestra.run.smithi151.stdout:Total 637 kB/s | 771 kB 00:01 2024-03-25T19:19:37.141 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:19:37.145 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:19:37.145 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:19:37.187 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:19:37.188 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:19:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:19:37.480 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:19:37.854 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:19:38.099 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:19:38.099 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:38.099 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:19:38.099 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:38.099 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:38.099 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:19:38.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:19:38.887 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:19:02 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:19:39.036 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:19:39.037 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:39.037 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:19:39.037 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:39.037 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:19:39.037 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:19:39.038 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:39.038 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:19:39.038 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:39.038 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:19:39.038 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:39.038 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 1.1 M 2024-03-25T19:19:39.038 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 4.3 M 2024-03-25T19:19:39.039 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:19:41.468 INFO:teuthology.orchestra.run.smithi151.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 474 kB/s | 1.1 MB 00:02 2024-03-25T19:19:41.469 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:41.469 INFO:teuthology.orchestra.run.smithi151.stdout:Total 473 kB/s | 1.1 MB 00:02 2024-03-25T19:19:41.469 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:19:41.473 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:19:41.473 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:19:41.517 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:19:41.518 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:19:41.737 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:19:41.842 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:19:42.301 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:19:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:19:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:42.557 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:19:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:42.557 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:19:42.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:19:43.340 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:19:07 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:19:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:19:43.490 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 337 k 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.0 M 2024-03-25T19:19:43.491 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:19:44.654 INFO:teuthology.orchestra.run.smithi151.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 290 kB/s | 337 kB 00:01 2024-03-25T19:19:44.655 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:44.655 INFO:teuthology.orchestra.run.smithi151.stdout:Total 290 kB/s | 337 kB 00:01 2024-03-25T19:19:44.656 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:19:44.660 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:19:44.660 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:19:44.698 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:19:44.699 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:19:44.840 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:19:44.953 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:19:45.297 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:19:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:19:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:45.558 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:19:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:45.558 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:19:45.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:19:46.351 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:19:10 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:19:46.499 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:19:46.499 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:46.499 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:19:46.499 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:46.499 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:19:46.499 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:19:46.499 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:46.500 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:19:46.500 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:46.500 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:19:46.500 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:46.500 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 509 k 2024-03-25T19:19:46.500 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.5 M 2024-03-25T19:19:46.500 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:19:48.374 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 272 kB/s | 509 kB 00:01 2024-03-25T19:19:48.375 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:48.375 INFO:teuthology.orchestra.run.smithi151.stdout:Total 271 kB/s | 509 kB 00:01 2024-03-25T19:19:48.375 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:19:48.379 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:19:48.379 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:19:48.421 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:19:48.421 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:19:48.690 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:19:49.058 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:19:49.324 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:19:49.324 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:49.324 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:19:49.324 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:49.324 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:49.324 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:19:49.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:19:50.094 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:19:14 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:19:50.242 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:19:50.243 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:19:50.243 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:50.243 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:19:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:19:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:50.243 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:19:50.243 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:19:50.243 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:19:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:19:50.244 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 52 M 2024-03-25T19:19:50.244 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 298 M 2024-03-25T19:19:50.244 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:20:15.809 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 414 kB/s | 52 MB 02:07 2024-03-25T19:20:15.809 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:20:15.810 INFO:teuthology.orchestra.run.smithi123.stdout:Total 414 kB/s | 52 MB 02:07 2024-03-25T19:20:15.810 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:20:15.813 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:20:15.813 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:20:16.149 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:20:16.149 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:20:22.050 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:20:22.374 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:20:24.361 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:20:24.616 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:20:24.616 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:20:24.616 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:20:24.616 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:20:24.616 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:20:24.616 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:20:24.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:20:25.362 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 1:19:51 ago on Mon 25 Mar 2024 06:00:34 PM UTC. 2024-03-25T19:20:25.509 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-25T19:20:25.510 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:20:25.510 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-25T19:20:25.510 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:20:25.510 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-25T19:20:25.510 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:20:25.510 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:20:25.510 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-25T19:20:25.510 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-25T19:20:25.510 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-25T19:20:25.510 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:20:25.511 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 1.3 M 2024-03-25T19:20:25.511 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 5.5 M 2024-03-25T19:20:25.511 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-25T19:20:28.019 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 511 kB/s | 1.3 MB 00:02 2024-03-25T19:20:28.019 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:20:28.020 INFO:teuthology.orchestra.run.smithi123.stdout:Total 511 kB/s | 1.3 MB 00:02 2024-03-25T19:20:28.020 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-25T19:20:28.024 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-25T19:20:28.024 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-25T19:20:28.069 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-25T19:20:28.069 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-25T19:20:28.326 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-25T19:20:28.426 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:20:28.800 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:20:29.053 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:20:29.053 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:20:29.053 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-25T19:20:29.053 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:20:29.054 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:20:29.054 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-25T19:20:29.226 DEBUG:teuthology.parallel:result is None 2024-03-25T19:21:19.732 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 590 kB/s | 52 MB 01:29 2024-03-25T19:21:19.733 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:19.733 INFO:teuthology.orchestra.run.smithi151.stdout:Total 590 kB/s | 52 MB 01:29 2024-03-25T19:21:19.733 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:21:19.737 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:21:19.737 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:21:20.070 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:21:20.071 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:21:25.976 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:21:26.255 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:21:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:21:28.736 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:21:28.736 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:28.736 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:21:28.736 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:28.736 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:28.736 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:21:28.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:21:29.507 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 1:20:53 ago on Mon 25 Mar 2024 06:00:36 PM UTC. 2024-03-25T19:21:29.656 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-25T19:21:29.656 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:21:29.656 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-25T19:21:29.657 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:21:29.657 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-25T19:21:29.657 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:21:29.657 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:29.657 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-25T19:21:29.657 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-25T19:21:29.657 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-25T19:21:29.657 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:29.658 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 1.3 M 2024-03-25T19:21:29.658 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 5.5 M 2024-03-25T19:21:29.658 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-25T19:21:33.008 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 383 kB/s | 1.3 MB 00:03 2024-03-25T19:21:33.009 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:33.009 INFO:teuthology.orchestra.run.smithi151.stdout:Total 382 kB/s | 1.3 MB 00:03 2024-03-25T19:21:33.010 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-25T19:21:33.014 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-25T19:21:33.014 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-25T19:21:33.060 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-25T19:21:33.060 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-25T19:21:33.298 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-25T19:21:33.398 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:21:33.820 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:21:34.083 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:21:34.083 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:34.083 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-25T19:21:34.083 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:34.083 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:34.083 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-25T19:21:34.279 DEBUG:teuthology.parallel:result is None 2024-03-25T19:21:34.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:21:34.423 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:21:34.447 INFO:teuthology.orchestra.run.smithi123.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:21:34.447 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:21:34.447 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:21:34.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:21:34.586 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:21:34.606 INFO:teuthology.orchestra.run.smithi151.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:21:34.606 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:21:34.606 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:21:34.607 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:21:34.608 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:21:34.608 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:21:34.639 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:21:34.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:21:34.674 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:21:34.674 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:21:34.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:21:34.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:21:34.781 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:21:34.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:21:34.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:21:34.874 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:21:34.874 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:21:34.875 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:21:34.905 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:21:34.973 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:21:34.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:21:35.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:21:35.067 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:21:35.067 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:21:35.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:21:35.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:21:35.160 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:21:35.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:21:35.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:21:35.254 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:21:35.334 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:21:35.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /var/log/ceph 2024-03-25T19:21:35.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /var/log/ceph 2024-03-25T19:21:35.363 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:21:35.363 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:21:35.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:21:35.430 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:21:35.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:21:35.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:21:35.507 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:21:35.507 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:21:35.507 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621312', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi123.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSVD7vP2KGEGEkXf8ZFIHPD1QPwQEUOKMkbsOPXii1QG6h3uRKB2zvn8oiBAn+z+4YhpYIGHVnkHe0aNLvWdS0=', 'smithi151.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUmqGh/fvs4G4Grt3FCYf2UIBDkVXoq+MdCOmetXyIl4A1NWCCJbh5kiJQcDb5hCwB04Fl8GbhBTVyZKB+7Tdg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:21:35.508 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:21:35.537 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:21:35.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:21:35.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:21:35.619 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:21:35.619 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:21:35.671 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:21:35.672 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-03-25T19:21:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:21:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:21:35.726 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:21:35.726 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:21:35.726 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:21:35.726 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-25 18:08:11.656881752 +0000 2024-03-25T19:21:35.726 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-25 17:59:25.700058058 +0000 2024-03-25T19:21:35.726 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-25 17:59:25.700058058 +0000 2024-03-25T19:21:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-25 17:59:25.700058058 +0000 2024-03-25T19:21:35.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:21:35.790 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-25T19:21:35.790 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-25T19:21:35.790 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000338349 s, 1.5 MB/s 2024-03-25T19:21:35.791 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:21:35.847 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-03-25T19:21:35.901 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:21:35.902 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:21:35.902 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:21:35.902 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:21:35.902 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:21:35.902 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-25 18:08:11.656881752 +0000 2024-03-25T19:21:35.902 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-25 17:59:26.029049789 +0000 2024-03-25T19:21:35.902 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-25 17:59:26.029049789 +0000 2024-03-25T19:21:35.902 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-25 17:59:26.029049789 +0000 2024-03-25T19:21:35.902 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:21:35.966 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-25T19:21:35.966 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-25T19:21:35.966 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.00018948 s, 2.7 MB/s 2024-03-25T19:21:35.968 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:21:36.022 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-03-25T19:21:36.078 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:21:36.078 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:21:36.078 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:21:36.078 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:21:36.078 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:21:36.078 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-25 18:08:11.657881727 +0000 2024-03-25T19:21:36.078 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-25 17:59:26.364041369 +0000 2024-03-25T19:21:36.078 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-25 17:59:26.364041369 +0000 2024-03-25T19:21:36.078 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-25 17:59:26.364041369 +0000 2024-03-25T19:21:36.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:21:36.142 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-25T19:21:36.143 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-25T19:21:36.143 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000164628 s, 3.1 MB/s 2024-03-25T19:21:36.144 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:21:36.199 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-03-25T19:21:36.253 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:21:36.254 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:21:36.254 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T19:21:36.254 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:21:36.254 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:21:36.254 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-25 18:08:11.657881727 +0000 2024-03-25T19:21:36.254 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-25 17:59:26.695033050 +0000 2024-03-25T19:21:36.254 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-25 17:59:26.695033050 +0000 2024-03-25T19:21:36.254 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-25 17:59:26.695033050 +0000 2024-03-25T19:21:36.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:21:36.319 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-25T19:21:36.319 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-25T19:21:36.319 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000195636 s, 2.6 MB/s 2024-03-25T19:21:36.320 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:21:36.375 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:21:36.376 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:21:36.376 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:21:36.393 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:21:36.393 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-03-25T19:21:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:21:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:21:36.449 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:21:36.449 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:21:36.450 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:21:36.450 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-25 18:06:51.369076914 +0000 2024-03-25T19:21:36.450 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-25 17:59:25.446144206 +0000 2024-03-25T19:21:36.450 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-25 17:59:25.446144206 +0000 2024-03-25T19:21:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-25 17:59:25.446144206 +0000 2024-03-25T19:21:36.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:21:36.518 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-25T19:21:36.518 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-25T19:21:36.518 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000282272 s, 1.8 MB/s 2024-03-25T19:21:36.519 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:21:36.577 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-03-25T19:21:36.634 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:21:36.634 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:21:36.634 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T19:21:36.634 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:21:36.634 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:21:36.634 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-25 18:06:51.369076914 +0000 2024-03-25T19:21:36.634 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-25 17:59:25.778140549 +0000 2024-03-25T19:21:36.634 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-25 17:59:25.778140549 +0000 2024-03-25T19:21:36.634 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-25 17:59:25.778140549 +0000 2024-03-25T19:21:36.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:21:36.701 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-25T19:21:36.701 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-25T19:21:36.701 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000217004 s, 2.4 MB/s 2024-03-25T19:21:36.702 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:21:36.759 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-03-25T19:21:36.815 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:21:36.815 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:21:36.815 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:21:36.815 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:21:36.815 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:21:36.815 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-25 18:06:51.370076903 +0000 2024-03-25T19:21:36.815 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-25 17:59:26.106136937 +0000 2024-03-25T19:21:36.815 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-25 17:59:26.106136937 +0000 2024-03-25T19:21:36.815 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-25 17:59:26.106136937 +0000 2024-03-25T19:21:36.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:21:36.881 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-25T19:21:36.881 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-25T19:21:36.881 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000193978 s, 2.6 MB/s 2024-03-25T19:21:36.882 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:21:36.938 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-03-25T19:21:36.993 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:21:36.994 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:21:36.994 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T19:21:36.994 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:21:36.994 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:21:36.994 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-25 18:06:51.370076903 +0000 2024-03-25T19:21:36.994 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-25 17:59:26.437133291 +0000 2024-03-25T19:21:36.994 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-25 17:59:26.437133291 +0000 2024-03-25T19:21:36.994 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-25 17:59:26.437133291 +0000 2024-03-25T19:21:36.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:21:37.059 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-25T19:21:37.059 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-25T19:21:37.060 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.00019068 s, 2.7 MB/s 2024-03-25T19:21:37.061 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:21:37.116 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:21:37.116 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:21:37.116 INFO:tasks.ceph:Generating config... 2024-03-25T19:21:37.117 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:21:37.117 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:21:37.117 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:21:37.118 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:21:37.118 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:21:37.118 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:21:37.118 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:21:37.118 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:21:37.118 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:21:37.118 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:21:37.118 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:21:37.118 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:21:37.118 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:21:37.118 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:21:37.118 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:21:37.119 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:21:37.119 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:21:37.119 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:21:37.119 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:21:37.119 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:21:37.119 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:21:37.119 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:21:37.119 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:21:37.119 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:21:37.119 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:21:37.119 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:21:37.119 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:21:37.119 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:21:37.119 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:21:37.119 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:21:37.119 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:21:37.120 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:21:37.121 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:21:37.121 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:21:37.121 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:21:37.121 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:21:37.121 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:21:37.121 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:21:37.121 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:21:37.121 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:21:37.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:21:37.177 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:21:37.180 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:21:37.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:21:37.250 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.123:6789]'), ('mon.c', '[v1:172.21.15.123:6790]'), ('mon.b', '[v1:172.21.15.151:6789]')] 2024-03-25T19:21:37.251 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.123:6789],[v1:172.21.15.123:6790],[v1:172.21.15.151:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:21:37.252 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:21:37.252 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:21:37.305 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.123:6789]' --addv c '[v1:172.21.15.123:6790]' --addv b '[v1:172.21.15.151:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:21:37.389 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:21:37.389 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: generated fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:setting min_mon_release = quincy 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 0 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:min_mon_release 17 (quincy) 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:election_strategy: 3 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:21:37.390 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:21:37.391 DEBUG:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:21:37.446 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6494055c-b81c-4348-a74d-41946ff3e6ce... 2024-03-25T19:21:37.447 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:21:37.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:21:37.535 INFO:teuthology.orchestra.run.smithi123.stdout:[global] 2024-03-25T19:21:37.535 INFO:teuthology.orchestra.run.smithi123.stdout: chdir = "" 2024-03-25T19:21:37.535 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:21:37.535 INFO:teuthology.orchestra.run.smithi123.stdout: auth supported = cephx 2024-03-25T19:21:37.535 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.535 INFO:teuthology.orchestra.run.smithi123.stdout: filestore xattr use omap = true 2024-03-25T19:21:37.535 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.535 INFO:teuthology.orchestra.run.smithi123.stdout: mon clock drift allowed = 1.000 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: osd crush chooseleaf type = 0 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: auth debug = true 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on old message = true 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on bug = true 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: mon pg warn max object skew = 0 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default size = 2 2024-03-25T19:21:37.536 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow primary affinity = true 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow pg remap = true 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on no sortbitwise = false 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on too few osds = false 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: mon_allow_pool_size_one = true 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: osd default data pool replay window = 5 2024-03-25T19:21:37.537 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: mon allow pool delete = true 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: mon cluster log file level = debug 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: debug asserts on shutdown = true 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: mon health detail to clog = false 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: mon host = "[v1:172.21.15.123:6789],[v1:172.21.15.123:6790],[v1:172.21.15.151:6789]" 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: mon client directed command retry = 5 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: mon election default strategy = 3 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: ms bind msgr2 = False 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: ms inject socket failures = 5000 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: ms type = async 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: osd heartbeat grace = 80 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: fsid = 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:21:37.538 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout:[osd] 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd journal size = 100 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub load threshold = 5.0 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub max interval = 600 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd recover clone overlap = true 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd recovery max chunk = 1048576 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug shutdown = true 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug op order = true 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify stray on activate = true 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug trim objects = true 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.539 INFO:teuthology.orchestra.run.smithi123.stdout: osd open classes on start = true 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug pg log writeout = true 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: osd map max advance = 10 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: journal zero on create = true 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: filestore apply finisher threads = 3 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: bdev debug aio = true 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug misdirected ops = true 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: bdev async discard = True 2024-03-25T19:21:37.540 INFO:teuthology.orchestra.run.smithi123.stdout: bdev enable discard = True 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore allocator = stupid 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore block size = 96636764160 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore fsck on mount = True 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore zero block detection = True 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluefs = 1 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluestore = 1 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: debug monc = 20 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: debug osd = 20 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: debug rocksdb = 10 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd full ratio = 0.9 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: osd class default list = * 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: osd class load list = * 2024-03-25T19:21:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: osd client watch timeout = 120 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify cached snaps = True 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify missing on start = True 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: osd fast shutdown = False 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: osd max markdown count = 1000 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: osd mclock override recovery settings = True 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: osd objectstore = bluestore 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: osd op queue = debug_random 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: osd op queue cut off = debug_random 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: osd shutdown pgref assert = True 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr] 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgr = 20 2024-03-25T19:21:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: mgr/telemetry/nag = false 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout:[mon] 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: debug paxos = 20 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: mon data avail warn = 5 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: mon mgr mkfs grace = 240 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd reporter subtree level = osd 2024-03-25T19:21:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd prime pg temp = true 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: # 1m isn't quite enough 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:21:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: mon min osdmap epochs = 50 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd crush smoke test = False 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: mon osdmap full prune interval = 2 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: mon osdmap full prune min = 15 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: mon scrub interval = 300 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on pool no app = False 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: paxos service trim min = 10 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout:[client] 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: rgw cache enabled = true 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable ops log = true 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable usage log = true 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:21:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: debug monc = 20 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: debug objecter = 20 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: debug rados = 20 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.a] 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.c] 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.b] 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi151.stdout: chdir = "" 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi151.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:21:37.546 INFO:teuthology.orchestra.run.smithi151.stdout: auth supported = cephx 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: filestore xattr use omap = true 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: mon clock drift allowed = 1.000 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: osd crush chooseleaf type = 0 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: auth debug = true 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on old message = true 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on bug = true 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: mon pg warn max object skew = 0 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.547 INFO:teuthology.orchestra.run.smithi151.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default size = 2 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow primary affinity = true 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow pg remap = true 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on no sortbitwise = false 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on too few osds = false 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:21:37.548 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: mon_allow_pool_size_one = true 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: osd default data pool replay window = 5 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: mon allow pool delete = true 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: mon cluster log file level = debug 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: debug asserts on shutdown = true 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: mon health detail to clog = false 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: mon host = "[v1:172.21.15.123:6789],[v1:172.21.15.123:6790],[v1:172.21.15.151:6789]" 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: mon client directed command retry = 5 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: mon election default strategy = 3 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: ms bind msgr2 = False 2024-03-25T19:21:37.549 INFO:teuthology.orchestra.run.smithi151.stdout: ms inject socket failures = 5000 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: ms type = async 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: osd heartbeat grace = 80 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout:[osd] 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: osd journal size = 100 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub load threshold = 5.0 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub max interval = 600 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: osd recover clone overlap = true 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: osd recovery max chunk = 1048576 2024-03-25T19:21:37.550 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug shutdown = true 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug op order = true 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify stray on activate = true 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug trim objects = true 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: osd open classes on start = true 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug pg log writeout = true 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: osd map max advance = 10 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: journal zero on create = true 2024-03-25T19:21:37.551 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: filestore apply finisher threads = 3 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: bdev debug aio = true 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug misdirected ops = true 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: bdev async discard = True 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: bdev enable discard = True 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore allocator = stupid 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore block size = 96636764160 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore fsck on mount = True 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore zero block detection = True 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluefs = 1 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluestore = 1 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: debug monc = 20 2024-03-25T19:21:37.552 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: debug osd = 20 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: debug rocksdb = 10 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd full ratio = 0.9 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: osd class default list = * 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: osd class load list = * 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: osd client watch timeout = 120 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify cached snaps = True 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify missing on start = True 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: osd fast shutdown = False 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: osd max markdown count = 1000 2024-03-25T19:21:37.553 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock override recovery settings = True 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: osd objectstore = bluestore 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: osd op queue = debug_random 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: osd op queue cut off = debug_random 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: osd shutdown pgref assert = True 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr] 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: debug mgr = 20 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: mgr/telemetry/nag = false 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.554 INFO:teuthology.orchestra.run.smithi151.stdout:[mon] 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: debug paxos = 20 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: mon data avail warn = 5 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: mon mgr mkfs grace = 240 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd reporter subtree level = osd 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd prime pg temp = true 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:21:37.555 INFO:teuthology.orchestra.run.smithi151.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: # 1m isn't quite enough 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: mon min osdmap epochs = 50 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd crush smoke test = False 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune interval = 2 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune min = 15 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:21:37.556 INFO:teuthology.orchestra.run.smithi151.stdout: mon scrub interval = 300 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on pool no app = False 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: paxos service trim min = 10 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout:[client] 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: rgw cache enabled = true 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable ops log = true 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable usage log = true 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: debug monc = 20 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: debug objecter = 20 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout: debug rados = 20 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.a] 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.c] 2024-03-25T19:21:37.557 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.b] 2024-03-25T19:21:37.559 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:21:37.559 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:21:37.613 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:21:37.613 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:21:37.613 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:21:37.628 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:21:37.628 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:21:37.682 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T19:21:37.682 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:21:37.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:21:37.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:21:37.761 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:21:37.761 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:21:37.816 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T19:21:37.816 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:21:37.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:21:37.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:21:37.856 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:21:37.857 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:21:37.911 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:21:37.911 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:21:37.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:21:37.966 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:21:37.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:21:38.025 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:21:38.028 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:21:38.028 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:21:38.028 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:21:38.070 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:21:38.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:21:38.127 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:21:38.143 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:21:38.143 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:21:38.143 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:21:38.170 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:21:38.170 INFO:tasks.ceph:role: osd.0 2024-03-25T19:21:38.170 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T19:21:38.170 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:21:38.243 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:21:38.243 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:21:38.244 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:21:38.244 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:21:38.244 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:21:38.244 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:21:38.244 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:21:38.244 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:21:38.244 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:21:38.244 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:21:38.765 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-03-25T19:21:38.767 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-03-25T19:21:38.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:21:38.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:21:38.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:21:38.992 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:21:38.992 INFO:tasks.ceph:role: osd.1 2024-03-25T19:21:38.992 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T19:21:38.992 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:21:39.065 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:21:39.066 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:21:39.066 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:21:39.066 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:21:39.066 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:21:39.066 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:21:39.066 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:21:39.066 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:21:39.066 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:21:39.066 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:21:39.559 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-03-25T19:21:39.561 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-03-25T19:21:39.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:21:39.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:21:39.683 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:21:39.751 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:21:39.751 INFO:tasks.ceph:role: osd.2 2024-03-25T19:21:39.751 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T19:21:39.751 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:21:39.825 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:21:39.825 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:21:39.825 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:21:39.825 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:21:39.825 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:21:39.825 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:21:39.825 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:21:39.825 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:21:39.825 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:21:39.825 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:21:40.343 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-03-25T19:21:40.345 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-03-25T19:21:40.345 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:21:40.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:21:40.522 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:21:40.590 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:21:40.590 INFO:tasks.ceph:role: osd.3 2024-03-25T19:21:40.590 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T19:21:40.590 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:21:40.664 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:21:40.664 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:21:40.664 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:21:40.664 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:21:40.664 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:21:40.664 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:21:40.664 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:21:40.665 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:21:40.665 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:21:40.665 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:21:41.181 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-03-25T19:21:41.183 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-03-25T19:21:41.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:21:41.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:21:41.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:21:41.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:41.404+0000 7ff6c421e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:21:41.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:41.405+0000 7ff6c421e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:21:41.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:41.405+0000 7ff6c421e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:21:41.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:41.405+0000 7ff6c421e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:21:41.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:41.405+0000 7ff6c421e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:21:41.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:41.405+0000 7ff6c421e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:21:44.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:21:44.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:21:44.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:44.361+0000 7facd2222640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:21:44.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:44.361+0000 7facd2222640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:21:44.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:44.361+0000 7facd2222640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:21:44.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:44.361+0000 7facd2222640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:21:44.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:44.361+0000 7facd2222640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:21:44.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:44.362+0000 7facd2222640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:21:47.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:21:47.242 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:21:47.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:47.337+0000 7f2991413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:21:47.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:47.337+0000 7f2991413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:21:47.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:47.337+0000 7f2991413640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:21:47.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:47.337+0000 7f2991413640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:21:47.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:47.337+0000 7f2991413640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:21:47.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:47.338+0000 7f2991413640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:21:50.181 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:21:50.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:21:50.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:50.301+0000 7fc446145640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:21:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:50.301+0000 7fc446145640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:21:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:50.301+0000 7fc446145640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:21:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:50.301+0000 7fc446145640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:21:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:50.302+0000 7fc446145640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:21:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:21:50.302+0000 7fc446145640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:21:53.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:21:53.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:21:53.198 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:21:53.199 INFO:tasks.ceph:role: osd.4 2024-03-25T19:21:53.199 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T19:21:53.199 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:21:53.276 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:21:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:21:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:21:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:21:53.276 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:21:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:21:53.276 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:21:53.276 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:21:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:21:53.276 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:21:53.797 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-25T19:21:53.799 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-25T19:21:53.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:21:53.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:21:53.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:21:54.053 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:21:54.053 INFO:tasks.ceph:role: osd.5 2024-03-25T19:21:54.053 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T19:21:54.053 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:21:54.129 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:21:54.129 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:21:54.129 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:21:54.129 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:21:54.129 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:21:54.129 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:21:54.129 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:21:54.130 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:21:54.130 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:21:54.130 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:21:54.578 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-25T19:21:54.580 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-25T19:21:54.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:21:54.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:21:54.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:21:54.778 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:21:54.778 INFO:tasks.ceph:role: osd.6 2024-03-25T19:21:54.778 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T19:21:54.778 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:21:54.855 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:21:54.856 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:21:54.856 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:21:54.856 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:21:54.856 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:21:54.856 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:21:54.856 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:21:54.856 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:21:54.856 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:21:54.856 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:21:55.351 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-25T19:21:55.353 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-25T19:21:55.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:21:55.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:21:55.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:21:55.619 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:21:55.619 INFO:tasks.ceph:role: osd.7 2024-03-25T19:21:55.619 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T19:21:55.619 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:21:55.696 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:21:55.696 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:21:55.696 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:21:55.696 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:21:55.696 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:21:55.696 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:21:55.696 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:21:55.696 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:21:55.696 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:21:55.696 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:21:56.212 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-25T19:21:56.215 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-25T19:21:56.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:21:56.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:21:56.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:21:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:56.497+0000 7ff0f2f93640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:21:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:56.498+0000 7ff0f2f93640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:21:56.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:56.498+0000 7ff0f2f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:21:56.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:56.498+0000 7ff0f2f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:21:56.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:56.498+0000 7ff0f2f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:21:56.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:56.498+0000 7ff0f2f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:21:59.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:21:59.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:21:59.472 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:59.472+0000 7f415c00d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:21:59.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:59.473+0000 7f415c00d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:21:59.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:59.473+0000 7f415c00d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:21:59.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:59.473+0000 7f415c00d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:21:59.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:59.473+0000 7f415c00d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:21:59.473 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:21:59.473+0000 7f415c00d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:22:02.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:22:02.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:22:02.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:02.431+0000 7f31d4c19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:22:02.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:02.432+0000 7f31d4c19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:22:02.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:02.432+0000 7f31d4c19640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:22:02.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:02.432+0000 7f31d4c19640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:22:02.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:02.432+0000 7f31d4c19640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:22:02.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:02.432+0000 7f31d4c19640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:22:05.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:22:05.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:22:05.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:05.433+0000 7f166c9dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:22:05.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:05.433+0000 7f166c9dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:22:05.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:05.433+0000 7f166c9dd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:22:05.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:05.433+0000 7f166c9dd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:22:05.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:05.433+0000 7f166c9dd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:22:05.434 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:05.434+0000 7f166c9dd640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:22:08.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:22:08.313 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:22:08.313 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:22:08.313 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:22:08.346 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:22:08.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:22:08.419 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:22:08.419 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:22:08.490 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:22:08.490 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:22:08.562 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:22:08.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:22:08.633 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:22:08.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:22:08.666 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:22:08.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:22:08.735 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:22:08.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:22:08.804 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:22:08.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:22:08.873 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:22:08.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:22:08.941 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:22:08.941 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:22:08.960 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:22:08.960 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:22:08.997 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:22:08.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:22:09.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:22:09.040 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.y] 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDBzgFmLdipORAANnB6NrspvFOeq14eF2ieNQ== 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.0] 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDFzgFm/zUuGBAApmtfWn7jPnydpj3ufU19zA== 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.1] 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDIzgFm8DebFRAA63dVpyGBvduyrcxaHEdOPw== 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.2] 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDLzgFm13gpFBAAvH+gK5dE1K159oOtQLh5Aw== 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.3] 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDOzgFmdhcJEhAAAcw0idRN1Ko0XlcRd4K7vA== 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.x] 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDCzgFmUbOdARAAg27X6CSTekc25atk6na76A== 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.4] 2024-03-25T19:22:09.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDUzgFmVca4HRAAHFYHmZSaFRrdjuj7l268JQ== 2024-03-25T19:22:09.042 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.5] 2024-03-25T19:22:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDXzgFmAC8/HBAAUwkMIy42UiIO31zjKnjc6w== 2024-03-25T19:22:09.042 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.6] 2024-03-25T19:22:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDazgFmPxzKGRAARf1WNBYZILZHg8dsEeIDEA== 2024-03-25T19:22:09.042 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.7] 2024-03-25T19:22:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDdzgFmQWfkGRAA4m7l/CJiJiQDrogn27gWvw== 2024-03-25T19:22:09.042 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-03-25T19:22:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDCzgFmlKlHBBAAyu/ntCoBGr8yt7OI7slPEA== 2024-03-25T19:22:09.042 INFO:teuthology.orchestra.run.smithi123.stdout:[client.1] 2024-03-25T19:22:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDCzgFm25qzBxAA/+K7kqKEXjUBHqIwIzqjEg== 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.y] 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDBzgFmLdipORAANnB6NrspvFOeq14eF2ieNQ== 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.0] 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDFzgFm/zUuGBAApmtfWn7jPnydpj3ufU19zA== 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.1] 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDIzgFm8DebFRAA63dVpyGBvduyrcxaHEdOPw== 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.2] 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDLzgFm13gpFBAAvH+gK5dE1K159oOtQLh5Aw== 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.3] 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDOzgFmdhcJEhAAAcw0idRN1Ko0XlcRd4K7vA== 2024-03-25T19:22:09.064 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.x] 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDCzgFmUbOdARAAg27X6CSTekc25atk6na76A== 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.4] 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDUzgFmVca4HRAAHFYHmZSaFRrdjuj7l268JQ== 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.5] 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDXzgFmAC8/HBAAUwkMIy42UiIO31zjKnjc6w== 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.6] 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDazgFmPxzKGRAARf1WNBYZILZHg8dsEeIDEA== 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.7] 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDdzgFmQWfkGRAA4m7l/CJiJiQDrogn27gWvw== 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDCzgFmlKlHBBAAyu/ntCoBGr8yt7OI7slPEA== 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout:[client.1] 2024-03-25T19:22:09.065 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQDCzgFm25qzBxAA/+K7kqKEXjUBHqIwIzqjEg== 2024-03-25T19:22:09.066 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:22:09.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:22:09.160 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.214 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.375 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:22:09.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:22:09.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:22:09.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:22:09.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:22:09.700 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:22:09.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:22:09.756 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:22:09.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:22:09.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:22:09.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:22:09.906 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:22:09.973 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:22:10.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:22:10.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:22:10.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:22:10.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:22:10.248 DEBUG:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:22:10.250 DEBUG:teuthology.orchestra.run.smithi151:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:22:10.302 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:22:10.302 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:22:10.302 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:22:10.303 DEBUG:teuthology.orchestra.run.smithi123:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:22:10.305 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:22:10.305 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:22:10.305 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:22:10.305 DEBUG:teuthology.orchestra.run.smithi123:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:22:10.307 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:22:10.307 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:22:10.308 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:22:10.308 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:22:10.345 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:22:10.345 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:22:10.345 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:22:10.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:22:10.348 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:22:10.348 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:22:10.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:22:10.350 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:22:10.350 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:22:10.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:22:10.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.451+0000 7efef372b640 10 monclient: get_monmap_and_config 2024-03-25T19:22:10.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.451+0000 7efef372b640 10 monclient: build_initial_monmap 2024-03-25T19:22:10.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.451+0000 7efef372b640 10 monclient: monmap: 2024-03-25T19:22:10.451 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 0 2024-03-25T19:22:10.452 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:10.452 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:22:10.452306+0000 2024-03-25T19:22:10.452 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:22:10.452306+0000 2024-03-25T19:22:10.452 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:10.452 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:10.452 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:10.452 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:10.452 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:10.452 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:10.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.451+0000 7efef372b640 1 Processor -- start 2024-03-25T19:22:10.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 1 -- start start 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 10 monclient: init 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 10 monclient(hunting): picked mon.noname-c con 0x7efeec0a9920 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 10 monclient(hunting): picked mon.noname-a con 0x7efeec0b45a0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 10 monclient(hunting): picked mon.noname-b con 0x7efeec146450 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec06b100 con 0x7efeec0b45a0 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec0b2940 con 0x7efeec0a9920 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec13fec0 con 0x7efeec146450 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef372b640 10 monclient(hunting): authenticate will time out at 5567.812710s 2024-03-25T19:22:10.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef0cc6640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efeec145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:10.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efeec14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:10.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.452+0000 7efef14c7640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efeec0a9920 legacy=0x7efeec13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:10.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.653+0000 7efef0cc6640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efeec145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:10.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.653+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efeec14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:10.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:10.653+0000 7efef14c7640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efeec0a9920 legacy=0x7efeec13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:11.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:11.054+0000 7efef0cc6640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efeec145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:11.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:11.054+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efeec14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:11.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:11.054+0000 7efef14c7640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efeec0a9920 legacy=0x7efeec13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:11.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:11.855+0000 7efef0cc6640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efeec145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:11.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:11.855+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efeec14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:11.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:11.855+0000 7efef14c7640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efeec0a9920 legacy=0x7efeec13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 10 monclient(hunting): tick 2024-03-25T19:22:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:22:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:22:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:22:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efeec14dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:22:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efeec0a9920 legacy=0x7efeec13f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:22:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efeec145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:22:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7efed0003680 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7efeec146450 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7efeec0b45a0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec06b100 con 0x7efeec0b45a0 2024-03-25T19:22:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec0b2940 con 0x7efed0003680 2024-03-25T19:22:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec13fec0 con 0x7efeec146450 2024-03-25T19:22:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efee2ffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.452+0000 7efef0cc6640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed00093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.453+0000 7efef14c7640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.453+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:13.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.654+0000 7efef0cc6640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed00093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:13.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.654+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:13.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:13.654+0000 7efef14c7640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:14.055+0000 7efef0cc6640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed00093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:14.055+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:14.055+0000 7efef14c7640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:14.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:14.856+0000 7efef0cc6640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed00093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:14.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:14.856+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:14.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:14.856+0000 7efef14c7640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:16.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 10 monclient(hunting): tick 2024-03-25T19:22:16.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:22:16.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:22:16.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed00093a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed000cba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7efeec0b45a0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7efed0003680 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7efeec146450 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0011350 con 0x7efeec0b45a0 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec06b100 con 0x7efed0003680 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec0b2940 con 0x7efeec146450 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efee2ffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efef14c7640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:16.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:16.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.453+0000 7efef0cc6640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:16.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.654+0000 7efef14c7640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:16.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.654+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:16.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:16.654+0000 7efef0cc6640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:17.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:17.055+0000 7efef14c7640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:17.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:17.055+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:17.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:17.055+0000 7efef0cc6640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:17.330 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:00:06.864 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:17.625 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:00:07.197 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:17.627 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:00:07.200 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:17.638 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:00:07.171 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:17.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:17.856+0000 7efef14c7640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6789/0 2024-03-25T19:22:17.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:17.856+0000 7efef1cc8640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.123:6790/0 2024-03-25T19:22:17.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:17.856+0000 7efef0cc6640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2024-03-25T19:22:17.946 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:00:07.518 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:17.948 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:00:07.520 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:19.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 10 monclient(hunting): tick 2024-03-25T19:22:19.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:22:19.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:22:19.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:22:19.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 1 -- >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:22:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0010360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:22:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 1 -- >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed000cba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:22:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7efeec146450 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7efeec0b45a0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7efed0003680 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec13fec0 con 0x7efeec0b45a0 2024-03-25T19:22:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0011350 con 0x7efed0003680 2024-03-25T19:22:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec06b100 con 0x7efeec146450 2024-03-25T19:22:19.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.453+0000 7efee2ffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:19.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.497+0000 7efef1cc8640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 0x7efed0010320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:47906/0 (socket says 172.21.15.123:47906) 2024-03-25T19:22:19.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:19.497+0000 7efef1cc8640 1 -- 172.21.15.123:0/1638314708 learned_addr learned my addr 172.21.15.123:0/1638314708 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:22:22.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 10 monclient(hunting): tick 2024-03-25T19:22:22.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.123:6790/0 conn(0x7efeec146450 legacy=0x7efed0010ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.151:6789/0 conn(0x7efed0003680 legacy=0x7efed0010320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.123:6789/0 conn(0x7efeec0b45a0 legacy=0x7efed0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7efed00093b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7efed00125d0 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7efed0010320 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec0b2940 con 0x7efed0010320 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0010860 con 0x7efed00093b0 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0005b80 con 0x7efed00125d0 2024-03-25T19:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:22.453+0000 7efee2ffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:25.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 10 monclient(hunting): tick 2024-03-25T19:22:25.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:22:25.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:22:25.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:22:25.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.123:6790/0 conn(0x7efed00125d0 legacy=0x7efed00149c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:25.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.151:6789/0 conn(0x7efed00093b0 legacy=0x7efed000cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:25.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.123:6789/0 conn(0x7efed0010320 legacy=0x7efed0010ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:25.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:22:25.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7efed0003ac0 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:25.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7efed00093b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:25.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7efed000f9f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:25.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0005db0 con 0x7efed000f9f0 2024-03-25T19:22:25.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed00056f0 con 0x7efed00093b0 2024-03-25T19:22:25.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0005920 con 0x7efed0003ac0 2024-03-25T19:22:25.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:25.453+0000 7efee2ffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 10 monclient(hunting): tick 2024-03-25T19:22:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:22:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:22:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:22:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.123:6790/0 conn(0x7efed0003ac0 legacy=0x7efed000f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.151:6789/0 conn(0x7efed00093b0 legacy=0x7efed000cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:28.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.123:6789/0 conn(0x7efed000f9f0 legacy=0x7efed0004f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:28.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:28.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7efed0010ab0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:28.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7efed0003ac0 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:28.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7efed00069b0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:28.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed000fe70 con 0x7efed00069b0 2024-03-25T19:22:28.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0010e90 con 0x7efed0010ab0 2024-03-25T19:22:28.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0011020 con 0x7efed0003ac0 2024-03-25T19:22:28.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:28.454+0000 7efee2ffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:31.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 10 monclient(hunting): tick 2024-03-25T19:22:31.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.123:6790/0 conn(0x7efed0003ac0 legacy=0x7efed000f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.151:6789/0 conn(0x7efed0010ab0 legacy=0x7efed00062a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.123:6789/0 conn(0x7efed00069b0 legacy=0x7efed0003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7efed00045b0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7efed0005100 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7efed0003ac0 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0006e30 con 0x7efed00045b0 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed00062a0 con 0x7efed0005100 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0006430 con 0x7efed0003ac0 2024-03-25T19:22:31.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:31.454+0000 7efee2ffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:33.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.628+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960110002 0 0) 0x7efed0006e30 con 0x7efed00045b0 2024-03-25T19:22:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.628+0000 7efee3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.628+0000 7efee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.628+0000 7efee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.628+0000 7efee3fff640 10 monclient(hunting): my global_id is 4101 2024-03-25T19:22:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.628+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed8003520 con 0x7efed00045b0 2024-03-25T19:22:33.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.671+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284234925 0 0) 0x7efed0006430 con 0x7efed0003ac0 2024-03-25T19:22:33.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.671+0000 7efee3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:33.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.671+0000 7efee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:33.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.671+0000 7efee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:33.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.671+0000 7efee3fff640 10 monclient(hunting): my global_id is 4106 2024-03-25T19:22:33.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.671+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed0006e30 con 0x7efed0003ac0 2024-03-25T19:22:33.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.711+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597473529 0 0) 0x7efed00062a0 con 0x7efed0005100 2024-03-25T19:22:33.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.711+0000 7efee3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:33.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.711+0000 7efee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:33.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.711+0000 7efee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:33.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.711+0000 7efee3fff640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:22:33.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.711+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed0006430 con 0x7efed0005100 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 329132311 0 0) 0x7efed8003520 con 0x7efed00045b0 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.123:6790/0 conn(0x7efed0003ac0 legacy=0x7efed0003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.151:6789/0 conn(0x7efed0005100 legacy=0x7efed000bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 1 monclient: found mon.noname-a 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0006660 con 0x7efed00045b0 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 10 monclient: _finish_auth 0 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:22:33.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7efee8004090 con 0x7efed00045b0 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efef372b640 5 monclient: authenticate success, global_id 4101 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efef372b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.721+0000 7efee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.722+0000 7efee3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.722+0000 7efee3fff640 10 monclient: dump: 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 1 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:33.723 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:33.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.722+0000 7efef372b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:33.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.727+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efee80041f0 con 0x7efed00045b0 2024-03-25T19:22:33.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.727+0000 7efee3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:33.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.727+0000 7efee3fff640 1 -- 172.21.15.123:0/1638314708 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7efee8004600 con 0x7efed00045b0 2024-03-25T19:22:33.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.727+0000 7efee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:33.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.727+0000 7efee3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:33.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.727+0000 7efee3fff640 10 monclient: dump: 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 1 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.727+0000 7efef372b640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:33.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.727+0000 7efef372b640 4 set_mon_vals no callback set 2024-03-25T19:22:33.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.728+0000 7efef372b640 10 monclient: shutdown 2024-03-25T19:22:33.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.728+0000 7efef372b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:33.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.728+0000 7efef372b640 1 -- 172.21.15.123:0/1638314708 >> v1:172.21.15.123:6789/0 conn(0x7efed00045b0 legacy=0x7efed00049f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:33.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.728+0000 7efef372b640 1 -- 172.21.15.123:0/1638314708 shutdown_connections 2024-03-25T19:22:33.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.728+0000 7efef372b640 1 -- 172.21.15.123:0/1638314708 >> 172.21.15.123:0/1638314708 conn(0x7efeec0b7f40 msgr2=0x7efeec0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:33.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.728+0000 7efef372b640 1 -- 172.21.15.123:0/1638314708 shutdown_connections 2024-03-25T19:22:33.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.728+0000 7efef372b640 1 -- 172.21.15.123:0/1638314708 wait complete. 2024-03-25T19:22:33.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.729+0000 7efef372b640 10 monclient: build_initial_monmap 2024-03-25T19:22:33.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.729+0000 7efef372b640 10 monclient: monmap: 2024-03-25T19:22:33.731 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 0 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 0.000000 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:created 0.000000 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 1 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.729+0000 7efef372b640 1 librados: starting msgr at 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.729+0000 7efef372b640 1 librados: starting objecter 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.729+0000 7efef372b640 1 Processor -- start 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.729+0000 7efef372b640 1 -- start start 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.729+0000 7efef372b640 1 librados: setting wanted keys 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.729+0000 7efef372b640 1 librados: calling monclient init 2024-03-25T19:22:33.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.729+0000 7efef372b640 10 monclient: init 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef372b640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef372b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef372b640 10 monclient(hunting): picked mon.noname-b con 0x7efed0003ac0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef372b640 10 monclient(hunting): picked mon.noname-a con 0x7efed00045b0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef372b640 10 monclient(hunting): picked mon.noname-c con 0x7efed0005100 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef372b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec1d72c0 con 0x7efed00045b0 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef372b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec1452c0 con 0x7efed0003ac0 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef372b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec1d7e60 con 0x7efed0005100 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef372b640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef372b640 10 monclient(hunting): authenticate will time out at 5591.090109s 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef1cc8640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7efed0005100 0x7efeec1e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:49674/0 (socket says 172.21.15.123:49674) 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efef1cc8640 1 -- 172.21.15.123:0/3234034201 learned_addr learned my addr 172.21.15.123:0/3234034201 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efee1ffb640 10 client.?.objecter ms_handle_connect 0x7efed0005100 2024-03-25T19:22:33.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.730+0000 7efee1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:33.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.748+0000 7efee1ffb640 10 client.?.objecter ms_handle_connect 0x7efed0003ac0 2024-03-25T19:22:33.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.748+0000 7efee1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:33.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.777+0000 7efee1ffb640 10 client.?.objecter ms_handle_connect 0x7efed00045b0 2024-03-25T19:22:33.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.777+0000 7efee1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:33.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.793+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623532856 0 0) 0x7efeec1d7e60 con 0x7efed0005100 2024-03-25T19:22:33.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.793+0000 7efee1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:33.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.793+0000 7efee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:33.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.793+0000 7efee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:33.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.793+0000 7efee1ffb640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:22:33.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.793+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed0006280 con 0x7efed0005100 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330361663 0 0) 0x7efeec1452c0 con 0x7efed0003ac0 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 10 monclient(hunting): my global_id is 4111 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeec1d7e60 con 0x7efed0003ac0 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264740462 0 0) 0x7efeec1d72c0 con 0x7efed00045b0 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 10 monclient(hunting): my global_id is 4110 2024-03-25T19:22:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.800+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeec1452c0 con 0x7efed00045b0 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.819+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2289140448 0 0) 0x7efeec1452c0 con 0x7efed00045b0 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.819+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeec1d72c0 con 0x7efed00045b0 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.819+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7efedc0041e0 con 0x7efed00045b0 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.819+0000 7efee1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.819+0000 7efee1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.819+0000 7efee1ffb640 10 monclient(hunting): dump: 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 1 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:33.820 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:33.820 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:33.820 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:33.820 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:33.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.819+0000 7efee1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1230814234 0 0) 0x7efeec1d72c0 con 0x7efed00045b0 2024-03-25T19:22:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 >> v1:172.21.15.123:6790/0 conn(0x7efed0005100 legacy=0x7efeec1e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 >> v1:172.21.15.151:6789/0 conn(0x7efed0003ac0 legacy=0x7efeec0abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:22:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 1 monclient: found mon.a 2024-03-25T19:22:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec1d7ff0 con 0x7efed00045b0 2024-03-25T19:22:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efef372b640 5 monclient: authenticate success, global_id 4110 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efef372b640 10 monclient: _renew_subs 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efef372b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efef372b640 1 -- 172.21.15.123:0/3234034201 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeec142470 con 0x7efed00045b0 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efef372b640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efef372b640 10 monclient: _renew_subs 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efef372b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efef372b640 1 -- 172.21.15.123:0/3234034201 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efeec145770 con 0x7efed00045b0 2024-03-25T19:22:33.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.908+0000 7efef372b640 1 librados: init done 2024-03-25T19:22:33.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.909+0000 7efef372b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:33.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.909+0000 7efef372b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:33.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.909+0000 7efef372b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:33.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.909+0000 7efef372b640 1 -- 172.21.15.123:0/3234034201 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeec145770 con 0x7efed00045b0 2024-03-25T19:22:33.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efedc0045f0 con 0x7efed00045b0 2024-03-25T19:22:33.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:33.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7efedc004a40 con 0x7efed00045b0 2024-03-25T19:22:33.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:33.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:33.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 10 monclient: dump: 2024-03-25T19:22:33.918 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 1 2024-03-25T19:22:33.918 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:33.918 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:33.918 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efedc004ff0 con 0x7efed00045b0 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3902640108 0 0) 0x7efedc006210 con 0x7efed00045b0 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 10 client.4110.objecter ms_dispatch 0x7efeec000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee37fe640 4 set_mon_vals no callback set 2024-03-25T19:22:33.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:33.918+0000 7efee1ffb640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-25T19:22:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.080+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efedc0067c0 con 0x7efed00045b0 2024-03-25T19:22:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.080+0000 7efee1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:34.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.080+0000 7efee1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:34.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.151+0000 7efef372b640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:22:34.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.151+0000 7efef372b640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:22:34.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.151+0000 7efef372b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:34.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.151+0000 7efef372b640 1 -- 172.21.15.123:0/3234034201 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7efeec145770 con 0x7efed00045b0 2024-03-25T19:22:34.304 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:34.304+0000 7f327d637040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:22:34.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.323+0000 7efee1ffb640 1 -- 172.21.15.123:0/3234034201 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7efedc0052a0 con 0x7efed00045b0 2024-03-25T19:22:34.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.323+0000 7efee1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:22:34.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.323+0000 7efee1ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:22:34.323 INFO:teuthology.orchestra.run.smithi123.stderr:adjusted tunables profile to default 2024-03-25T19:22:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 10 librados: watch_flush enter 2024-03-25T19:22:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 10 librados: watch_flush exit 2024-03-25T19:22:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-25T19:22:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 10 client.4110.objecter successfully canceled tick 2024-03-25T19:22:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 10 monclient: shutdown 2024-03-25T19:22:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 1 -- 172.21.15.123:0/3234034201 >> v1:172.21.15.123:6789/0 conn(0x7efed00045b0 legacy=0x7efeec1d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 1 -- 172.21.15.123:0/3234034201 shutdown_connections 2024-03-25T19:22:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 1 -- 172.21.15.123:0/3234034201 >> 172.21.15.123:0/3234034201 conn(0x7efeec0b7f40 msgr2=0x7efeec0b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 1 -- 172.21.15.123:0/3234034201 shutdown_connections 2024-03-25T19:22:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 1 -- 172.21.15.123:0/3234034201 wait complete. 2024-03-25T19:22:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:34.324+0000 7efef372b640 1 librados: shutdown 2024-03-25T19:22:34.335 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:22:34.335 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:22:34.335 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:22:34.335 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:22:34.359 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:34.359+0000 7f327d637040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:22:34.362 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:22:34.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:22:34.430 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:22:34.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:22:34.498 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-25T19:22:34.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:22:34.510 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:34.510+0000 7f866c810040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:22:34.537 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:22:34.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:22:34.564 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:22:34.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:22:34.580 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:34.580+0000 7f866c810040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:22:34.632 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:22:34.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:22:34.702 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-25T19:22:34.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:22:34.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 4c17b60d-1ca6-4aea-93f8-fb79f8a41266 0 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.910+0000 7fe90ec63640 10 monclient: get_monmap_and_config 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.910+0000 7fe90ec63640 10 monclient: build_initial_monmap 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.910+0000 7fe90ec63640 10 monclient: monmap: 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:22:34.911062+0000 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:22:34.911062+0000 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:34.911 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:34.911 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 1 Processor -- start 2024-03-25T19:22:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 1 -- start start 2024-03-25T19:22:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 10 monclient: init 2024-03-25T19:22:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 10 monclient(hunting): picked mon.noname-a con 0x7fe9080b4700 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 10 monclient(hunting): picked mon.noname-b con 0x7fe908142a20 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 10 monclient(hunting): picked mon.noname-c con 0x7fe9080a96b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9080a9d60 con 0x7fe9080b4700 2024-03-25T19:22:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90813ff10 con 0x7fe9080a96b0 2024-03-25T19:22:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9080aa1f0 con 0x7fe908142a20 2024-03-25T19:22:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.911+0000 7fe90ec63640 10 monclient(hunting): authenticate will time out at 5586.195374s 2024-03-25T19:22:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.912+0000 7fe90c9ff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe9080b4700 0x7fe908142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.151:36330/0 (socket says 172.21.15.151:36330) 2024-03-25T19:22:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.912+0000 7fe90c9ff640 1 -- 172.21.15.151:0/3165307191 learned_addr learned my addr 172.21.15.151:0/3165307191 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013101153 0 0) 0x7fe9080a9d60 con 0x7fe9080b4700 2024-03-25T19:22:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:22:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f0003520 con 0x7fe9080b4700 2024-03-25T19:22:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964823611 0 0) 0x7fe9080aa1f0 con 0x7fe908142a20 2024-03-25T19:22:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:22:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.914+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9080a9d60 con 0x7fe908142a20 2024-03-25T19:22:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.915+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736336290 0 0) 0x7fe90813ff10 con 0x7fe9080a96b0 2024-03-25T19:22:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.915+0000 7fe8ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.915+0000 7fe8ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.915+0000 7fe8ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:34.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.915+0000 7fe8ff7fe640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:22:34.916 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.915+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9080aa1f0 con 0x7fe9080a96b0 2024-03-25T19:22:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735933777 0 0) 0x7fe8f0003520 con 0x7fe9080b4700 2024-03-25T19:22:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 >> v1:172.21.15.123:6790/0 conn(0x7fe908142a20 legacy=0x7fe90813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 >> v1:172.21.15.151:6789/0 conn(0x7fe9080a96b0 legacy=0x7fe90813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:22:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 1 monclient: found mon.noname-a 2024-03-25T19:22:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9080b4ae0 con 0x7fe9080b4700 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 10 monclient: _finish_auth 0 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe904001a10 con 0x7fe9080b4700 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe90ec63640 5 monclient: authenticate success, global_id 4116 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe90ec63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe8ff7fe640 10 monclient: dump: 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:34.920 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.918+0000 7fe90ec63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.920+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe904001b70 con 0x7fe9080b4700 2024-03-25T19:22:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.920+0000 7fe8ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.920+0000 7fe8ff7fe640 1 -- 172.21.15.151:0/3165307191 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe904001f80 con 0x7fe9080b4700 2024-03-25T19:22:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.920+0000 7fe90ec63640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.920+0000 7fe90ec63640 4 set_mon_vals no callback set 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.920+0000 7fe8ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.920+0000 7fe8ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.920+0000 7fe8ff7fe640 10 monclient: dump: 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 10 monclient: shutdown 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 1 -- 172.21.15.151:0/3165307191 >> v1:172.21.15.123:6789/0 conn(0x7fe9080b4700 legacy=0x7fe908142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 1 -- 172.21.15.151:0/3165307191 shutdown_connections 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 1 -- 172.21.15.151:0/3165307191 >> 172.21.15.151:0/3165307191 conn(0x7fe9080668e0 msgr2=0x7fe908066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 1 -- 172.21.15.151:0/3165307191 shutdown_connections 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 1 -- 172.21.15.151:0/3165307191 wait complete. 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 10 monclient: build_initial_monmap 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 10 monclient: monmap: 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-25T19:22:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 1 librados: starting msgr at 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.921+0000 7fe90ec63640 1 librados: starting objecter 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 1 Processor -- start 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 1 -- start start 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 1 librados: setting wanted keys 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 1 librados: calling monclient init 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 10 monclient: init 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 10 monclient(hunting): picked mon.noname-b con 0x7fe9080a96b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 10 monclient(hunting): picked mon.noname-a con 0x7fe9080b4700 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 10 monclient(hunting): picked mon.noname-c con 0x7fe908142a20 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9080abf00 con 0x7fe9080b4700 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9081d4fe0 con 0x7fe9080a96b0 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9080ac450 con 0x7fe908142a20 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.922+0000 7fe90ec63640 10 monclient(hunting): authenticate will time out at 5586.206392s 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.923+0000 7fe8fffff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe9080b4700 0x7fe9081db2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.151:36342/0 (socket says 172.21.15.151:36342) 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.923+0000 7fe8fffff640 1 -- 172.21.15.151:0/1363640550 learned_addr learned my addr 172.21.15.151:0/1363640550 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.923+0000 7fe8fd7fa640 10 client.?.objecter ms_handle_connect 0x7fe9080b4700 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.923+0000 7fe8fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.923+0000 7fe8fd7fa640 10 client.?.objecter ms_handle_connect 0x7fe9080a96b0 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.923+0000 7fe8fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.923+0000 7fe8fd7fa640 10 client.?.objecter ms_handle_connect 0x7fe908142a20 2024-03-25T19:22:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.923+0000 7fe8fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:34.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772306128 0 0) 0x7fe9080abf00 con 0x7fe9080b4700 2024-03-25T19:22:34.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:22:34.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:34.928 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f4004eb0 con 0x7fe9080b4700 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359357537 0 0) 0x7fe9080ac450 con 0x7fe908142a20 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9080abf00 con 0x7fe908142a20 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293384427 0 0) 0x7fe9081d4fe0 con 0x7fe9080a96b0 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:22:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.925+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9080ac450 con 0x7fe9080a96b0 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1153110196 0 0) 0x7fe8f4004eb0 con 0x7fe9080b4700 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9081d4fe0 con 0x7fe9080b4700 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe8f8001a30 con 0x7fe9080b4700 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 10 monclient(hunting): dump: 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 137400508 0 0) 0x7fe9080abf00 con 0x7fe908142a20 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8f4004eb0 con 0x7fe908142a20 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe9000017d0 con 0x7fe908142a20 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.930+0000 7fe8fd7fa640 10 monclient(hunting): dump: 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.931+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72999455 0 0) 0x7fe9080ac450 con 0x7fe9080a96b0 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.931+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9080abf00 con 0x7fe9080a96b0 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.931+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe904001e80 con 0x7fe9080a96b0 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.931+0000 7fe8fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.931+0000 7fe8fd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.931+0000 7fe8fd7fa640 10 monclient(hunting): dump: 2024-03-25T19:22:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:34.934 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:34.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3517452459 0 0) 0x7fe9081d4fe0 con 0x7fe9080b4700 2024-03-25T19:22:34.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 >> v1:172.21.15.123:6790/0 conn(0x7fe908142a20 legacy=0x7fe9081e1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 >> v1:172.21.15.151:6789/0 conn(0x7fe9080a96b0 legacy=0x7fe9081dabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:22:34.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 1 monclient: found mon.a 2024-03-25T19:22:34.935 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9080ac5e0 con 0x7fe9080b4700 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe8fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe90ec63640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe90ec63640 10 monclient: _renew_subs 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe90ec63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe90ec63640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9080ab880 con 0x7fe9080b4700 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe90ec63640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe90ec63640 10 monclient: _renew_subs 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe90ec63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe90ec63640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9081e33b0 con 0x7fe9080b4700 2024-03-25T19:22:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.933+0000 7fe90ec63640 1 librados: init done 2024-03-25T19:22:34.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.934+0000 7fe90ec63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:34.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.934+0000 7fe90ec63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:34.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.934+0000 7fe90ec63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:34.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:34.934+0000 7fe90ec63640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9080ac7d0 con 0x7fe9080b4700 2024-03-25T19:22:34.946 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:34.946+0000 7f327d637040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:22:35.000 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:35.000+0000 7f327d637040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:22:35.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.034+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8f8001c70 con 0x7fe9080b4700 2024-03-25T19:22:35.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.034+0000 7fe8fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:35.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.034+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe8f8003a60 con 0x7fe9080b4700 2024-03-25T19:22:35.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.034+0000 7fe8fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.034+0000 7fe8fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:35.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.034+0000 7fe8fd7fa640 10 monclient: dump: 2024-03-25T19:22:35.034 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.034 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.034 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.035 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.034+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe8f8003fd0 con 0x7fe9080b4700 2024-03-25T19:22:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.034+0000 7fe8feffd640 4 set_mon_vals no callback set 2024-03-25T19:22:35.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.035+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2302265466 0 0) 0x7fe8f80065d0 con 0x7fe9080b4700 2024-03-25T19:22:35.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.035+0000 7fe8fd7fa640 10 client.4119.objecter ms_dispatch 0x7fe908000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:22:35.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.035+0000 7fe8fd7fa640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:22:35.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.035+0000 7fe8fd7fa640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:22:35.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.035+0000 7fe8fd7fa640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:22:35.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.035+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe8f8006940 con 0x7fe9080b4700 2024-03-25T19:22:35.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.035+0000 7fe8fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:35.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.035+0000 7fe8fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:35.045 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:35.045+0000 7f327d637040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:22:35.099 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:35.099+0000 7f327d637040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:22:35.105 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.106+0000 7fe90ec63640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4c17b60d-1ca6-4aea-93f8-fb79f8a41266", "id": 0}] 2024-03-25T19:22:35.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.106+0000 7fe90ec63640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4c17b60d-1ca6-4aea-93f8-fb79f8a41266", "id": 0}] 2024-03-25T19:22:35.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.106+0000 7fe90ec63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:35.106 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.106+0000 7fe90ec63640 1 -- 172.21.15.151:0/1363640550 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4c17b60d-1ca6-4aea-93f8-fb79f8a41266", "id": 0} v 0) v1 -- 0x7fe9080adbc0 con 0x7fe9080b4700 2024-03-25T19:22:35.141 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:35.142+0000 7f327d637040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:22:35.177 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:35.177+0000 7f866c810040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:22:35.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.192+0000 7fe8fd7fa640 1 -- 172.21.15.151:0/1363640550 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4c17b60d-1ca6-4aea-93f8-fb79f8a41266", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2880088037 0 732707515) 0x7fe8f8006b60 con 0x7fe9080b4700 2024-03-25T19:22:35.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.192+0000 7fe8fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4c17b60d-1ca6-4aea-93f8-fb79f8a41266", "id": 0}] 2024-03-25T19:22:35.192 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.192+0000 7fe8fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:35.193 INFO:teuthology.orchestra.run.smithi151.stdout:0 2024-03-25T19:22:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 10 librados: watch_flush enter 2024-03-25T19:22:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 10 librados: watch_flush exit 2024-03-25T19:22:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:22:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:22:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 10 monclient: shutdown 2024-03-25T19:22:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 1 -- 172.21.15.151:0/1363640550 >> v1:172.21.15.123:6789/0 conn(0x7fe9080b4700 legacy=0x7fe9081db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 1 -- 172.21.15.151:0/1363640550 shutdown_connections 2024-03-25T19:22:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 1 -- 172.21.15.151:0/1363640550 >> 172.21.15.151:0/1363640550 conn(0x7fe9080668e0 msgr2=0x7fe9080b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:35.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 1 -- 172.21.15.151:0/1363640550 shutdown_connections 2024-03-25T19:22:35.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 1 -- 172.21.15.151:0/1363640550 wait complete. 2024-03-25T19:22:35.195 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.193+0000 7fe90ec63640 1 librados: shutdown 2024-03-25T19:22:35.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 6ebf8ea1-3898-4326-8acc-ea0056f287b0 1 2024-03-25T19:22:35.234 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:35.235+0000 7f866c810040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:22:35.265 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:35.266+0000 7f327d637040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:22:35.283 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:35.283+0000 7f866c810040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:22:35.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.304+0000 7f6687a1e640 10 monclient: get_monmap_and_config 2024-03-25T19:22:35.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.304+0000 7f6687a1e640 10 monclient: build_initial_monmap 2024-03-25T19:22:35.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.304+0000 7f6687a1e640 10 monclient: monmap: 2024-03-25T19:22:35.304 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:35.304 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.304 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:22:35.305088+0000 2024-03-25T19:22:35.304 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:22:35.305088+0000 2024-03-25T19:22:35.304 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:35.305 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.305 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:35.305 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:35.305 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:35.305 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.305 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.304+0000 7f6687a1e640 1 Processor -- start 2024-03-25T19:22:35.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.304+0000 7f6687a1e640 1 -- start start 2024-03-25T19:22:35.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.304+0000 7f6687a1e640 10 monclient: init 2024-03-25T19:22:35.306 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.305+0000 7f6687a1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.305+0000 7f6687a1e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.305+0000 7f6687a1e640 10 monclient(hunting): picked mon.noname-b con 0x7f66800b4700 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.305+0000 7f6687a1e640 10 monclient(hunting): picked mon.noname-c con 0x7f6680142a20 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.305+0000 7f6687a1e640 10 monclient(hunting): picked mon.noname-a con 0x7f66800a96b0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.305+0000 7f6687a1e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66800a9d60 con 0x7f66800a96b0 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.305+0000 7f6687a1e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668013ff10 con 0x7f6680142a20 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.305+0000 7f6687a1e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66800aa1f0 con 0x7f66800b4700 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.305+0000 7f6687a1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.305+0000 7f6687a1e640 10 monclient(hunting): authenticate will time out at 5586.589119s 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.306+0000 7f6685fbb640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f66800a96b0 0x7f668013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.151:36346/0 (socket says 172.21.15.151:36346) 2024-03-25T19:22:35.307 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.306+0000 7f6685fbb640 1 -- 172.21.15.151:0/2528133592 learned_addr learned my addr 172.21.15.151:0/2528133592 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:35.308 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600248166 0 0) 0x7f66800a9d60 con 0x7f66800a96b0 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6668003520 con 0x7f66800a96b0 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074674048 0 0) 0x7f668013ff10 con 0x7f6680142a20 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.308+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66800a9d60 con 0x7f6680142a20 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.309+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914428728 0 0) 0x7f66800aa1f0 con 0x7f66800b4700 2024-03-25T19:22:35.309 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.309+0000 7f667ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.309+0000 7f667ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.309+0000 7f667ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.309+0000 7f667ffff640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:22:35.310 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.309+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f668013ff10 con 0x7f66800b4700 2024-03-25T19:22:35.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897939662 0 0) 0x7f6668003520 con 0x7f66800a96b0 2024-03-25T19:22:35.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 >> v1:172.21.15.123:6790/0 conn(0x7f66800b4700 legacy=0x7f6680142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 >> v1:172.21.15.151:6789/0 conn(0x7f6680142a20 legacy=0x7f668013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.311 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 1 monclient: found mon.noname-a 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66800b4ae0 con 0x7f66800a96b0 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 10 monclient: _finish_auth 0 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6670001a10 con 0x7f66800a96b0 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f667ffff640 10 monclient: dump: 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.312 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.313 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.313 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.313 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.313 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.313 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.313 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.313 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.313 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f6687a1e640 5 monclient: authenticate success, global_id 4122 2024-03-25T19:22:35.313 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.311+0000 7f6687a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:35.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.313+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6670001b70 con 0x7f66800a96b0 2024-03-25T19:22:35.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.313+0000 7f667ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:35.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.313+0000 7f667ffff640 1 -- 172.21.15.151:0/2528133592 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6670001f80 con 0x7f66800a96b0 2024-03-25T19:22:35.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.313+0000 7f667ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.313+0000 7f667ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.313+0000 7f667ffff640 10 monclient: dump: 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.313+0000 7f6687a1e640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.313+0000 7f6687a1e640 4 set_mon_vals no callback set 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.314+0000 7f6687a1e640 10 monclient: shutdown 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.314+0000 7f6687a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.314+0000 7f6687a1e640 1 -- 172.21.15.151:0/2528133592 >> v1:172.21.15.123:6789/0 conn(0x7f66800a96b0 legacy=0x7f668013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.314+0000 7f6687a1e640 1 -- 172.21.15.151:0/2528133592 shutdown_connections 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.314+0000 7f6687a1e640 1 -- 172.21.15.151:0/2528133592 >> 172.21.15.151:0/2528133592 conn(0x7f66800668e0 msgr2=0x7f6680066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.314+0000 7f6687a1e640 1 -- 172.21.15.151:0/2528133592 shutdown_connections 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.314+0000 7f6687a1e640 1 -- 172.21.15.151:0/2528133592 wait complete. 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.315+0000 7f6687a1e640 10 monclient: build_initial_monmap 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.315+0000 7f6687a1e640 10 monclient: monmap: 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2024-03-25T19:22:35.317 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.315+0000 7f6687a1e640 1 librados: starting msgr at 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.315+0000 7f6687a1e640 1 librados: starting objecter 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.315+0000 7f6687a1e640 1 Processor -- start 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.315+0000 7f6687a1e640 1 -- start start 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.315+0000 7f6687a1e640 1 librados: setting wanted keys 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.315+0000 7f6687a1e640 1 librados: calling monclient init 2024-03-25T19:22:35.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.315+0000 7f6687a1e640 10 monclient: init 2024-03-25T19:22:35.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.316+0000 7f6687a1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.316+0000 7f6687a1e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.316+0000 7f6687a1e640 10 monclient(hunting): picked mon.noname-c con 0x7f66800a96b0 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.316+0000 7f6687a1e640 10 monclient(hunting): picked mon.noname-b con 0x7f66800b4700 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.316+0000 7f6687a1e640 10 monclient(hunting): picked mon.noname-a con 0x7f6680142a20 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.316+0000 7f6687a1e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66800aba20 con 0x7f6680142a20 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.316+0000 7f6687a1e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66800abd00 con 0x7f66800b4700 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.316+0000 7f6687a1e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668013bd80 con 0x7f66800a96b0 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.316+0000 7f6687a1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.316+0000 7f6687a1e640 10 monclient(hunting): authenticate will time out at 5586.600373s 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.317+0000 7f6685fbb640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f6680142a20 0x7f66801e2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.151:36354/0 (socket says 172.21.15.151:36354) 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.317+0000 7f6685fbb640 1 -- 172.21.15.151:0/3917087556 learned_addr learned my addr 172.21.15.151:0/3917087556 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.317+0000 7f667dffb640 10 client.?.objecter ms_handle_connect 0x7f6680142a20 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.317+0000 7f667dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:35.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.317+0000 7f667dffb640 10 client.?.objecter ms_handle_connect 0x7f66800a96b0 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.317+0000 7f667dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.317+0000 7f667dffb640 10 client.?.objecter ms_handle_connect 0x7f66800b4700 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.317+0000 7f667dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181210468 0 0) 0x7f66800aba20 con 0x7f6680142a20 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f666c004eb0 con 0x7f6680142a20 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195823336 0 0) 0x7f66800abd00 con 0x7f66800b4700 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:22:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66800aba20 con 0x7f66800b4700 2024-03-25T19:22:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248266970 0 0) 0x7f668013bd80 con 0x7f66800a96b0 2024-03-25T19:22:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:22:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.319+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66800abd00 con 0x7f66800a96b0 2024-03-25T19:22:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.323+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 558526398 0 0) 0x7f666c004eb0 con 0x7f6680142a20 2024-03-25T19:22:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.323+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f668013bd80 con 0x7f6680142a20 2024-03-25T19:22:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.323+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6670001ec0 con 0x7f6680142a20 2024-03-25T19:22:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.323+0000 7f667dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.323+0000 7f667dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.323+0000 7f667dffb640 10 monclient(hunting): dump: 2024-03-25T19:22:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.323+0000 7f667dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.324+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 715833744 0 0) 0x7f66800aba20 con 0x7f66800b4700 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.324+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f666c004eb0 con 0x7f66800b4700 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.324+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6674001970 con 0x7f66800b4700 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.324+0000 7f667dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.324+0000 7f667dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.324+0000 7f667dffb640 10 monclient(hunting): dump: 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.326 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.325+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678271203 0 0) 0x7f66800abd00 con 0x7f66800a96b0 2024-03-25T19:22:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.325+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66800aba20 con 0x7f66800a96b0 2024-03-25T19:22:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.325+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6678001830 con 0x7f66800a96b0 2024-03-25T19:22:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.325+0000 7f667dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.325+0000 7f667dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.325+0000 7f667dffb640 10 monclient(hunting): dump: 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.326+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 128966110 0 0) 0x7f668013bd80 con 0x7f6680142a20 2024-03-25T19:22:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 >> v1:172.21.15.123:6790/0 conn(0x7f66800a96b0 legacy=0x7f66800ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 >> v1:172.21.15.151:6789/0 conn(0x7f66800b4700 legacy=0x7f66801392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f667dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f667dffb640 1 monclient: found mon.a 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f667dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f667dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668013ccb0 con 0x7f6680142a20 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f667dffb640 10 monclient: _finish_auth 0 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f667dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f667dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f6687a1e640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f6687a1e640 10 monclient: _renew_subs 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f6687a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f6687a1e640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668013ac10 con 0x7f6680142a20 2024-03-25T19:22:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f6687a1e640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:35.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f6687a1e640 10 monclient: _renew_subs 2024-03-25T19:22:35.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f6687a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:35.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f6687a1e640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f668013b8d0 con 0x7f6680142a20 2024-03-25T19:22:35.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.327+0000 7f6687a1e640 1 librados: init done 2024-03-25T19:22:35.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.328+0000 7f6687a1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:35.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.328+0000 7f6687a1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:35.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.328+0000 7f6687a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:35.331 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.328+0000 7f6687a1e640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668013bbd0 con 0x7f6680142a20 2024-03-25T19:22:35.342 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:35.342+0000 7f327d637040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:22:35.342 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:35.343+0000 7f866c810040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:22:35.386 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:35.386+0000 7f866c810040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:22:35.388 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:35.388+0000 7f327d637040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:22:35.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.424+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6670004e10 con 0x7f6680142a20 2024-03-25T19:22:35.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.424+0000 7f667dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:35.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.424+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f66700015d0 con 0x7f6680142a20 2024-03-25T19:22:35.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.424+0000 7f667dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.424+0000 7f667dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:35.424 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.424+0000 7f667dffb640 10 monclient: dump: 2024-03-25T19:22:35.424 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.424 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.424 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.425 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.425 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.425 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.425 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.425 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.425 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.425 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.424+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f66700054b0 con 0x7f6680142a20 2024-03-25T19:22:35.425 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.424+0000 7f667f7fe640 4 set_mon_vals no callback set 2024-03-25T19:22:35.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.425+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3896507325 0 0) 0x7f66700069b0 con 0x7f6680142a20 2024-03-25T19:22:35.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.425+0000 7f667dffb640 10 client.4125.objecter ms_dispatch 0x7f6680000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:22:35.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.425+0000 7f667dffb640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:22:35.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.425+0000 7f667dffb640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:22:35.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.425+0000 7f667dffb640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:22:35.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.425+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6670005a00 con 0x7f6680142a20 2024-03-25T19:22:35.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.425+0000 7f667dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:35.426 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.425+0000 7f667dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:35.442 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:35.442+0000 7f327d637040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:22:35.492 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:35.492+0000 7f327d637040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:22:35.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.498+0000 7f6687a1e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6ebf8ea1-3898-4326-8acc-ea0056f287b0", "id": 1}] 2024-03-25T19:22:35.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.498+0000 7f6687a1e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6ebf8ea1-3898-4326-8acc-ea0056f287b0", "id": 1}] 2024-03-25T19:22:35.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.498+0000 7f6687a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:35.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.498+0000 7f6687a1e640 1 -- 172.21.15.151:0/3917087556 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6ebf8ea1-3898-4326-8acc-ea0056f287b0", "id": 1} v 0) v1 -- 0x7f668013bf10 con 0x7f6680142a20 2024-03-25T19:22:35.505 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:35.505+0000 7f866c810040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:22:35.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.535+0000 7f667dffb640 1 -- 172.21.15.151:0/3917087556 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ebf8ea1-3898-4326-8acc-ea0056f287b0", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 576898966 0 940483276) 0x7f6670005d70 con 0x7f6680142a20 2024-03-25T19:22:35.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.535+0000 7f667dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6ebf8ea1-3898-4326-8acc-ea0056f287b0", "id": 1}] 2024-03-25T19:22:35.535 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.535+0000 7f667dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:35.535 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 10 librados: watch_flush enter 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 10 librados: watch_flush exit 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 10 monclient: shutdown 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 1 -- 172.21.15.151:0/3917087556 >> v1:172.21.15.123:6789/0 conn(0x7f6680142a20 legacy=0x7f66801e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 1 -- 172.21.15.151:0/3917087556 shutdown_connections 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 1 -- 172.21.15.151:0/3917087556 >> 172.21.15.151:0/3917087556 conn(0x7f66800668e0 msgr2=0x7f668013d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 1 -- 172.21.15.151:0/3917087556 shutdown_connections 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 1 -- 172.21.15.151:0/3917087556 wait complete. 2024-03-25T19:22:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.536+0000 7f6687a1e640 1 librados: shutdown 2024-03-25T19:22:35.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 96cf3100-b688-4a78-87ef-06b8a6806ac3 2 2024-03-25T19:22:35.584 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:35.584+0000 7f866c810040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:22:35.629 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:35.629+0000 7f866c810040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:22:35.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.646+0000 7f06a413d640 10 monclient: get_monmap_and_config 2024-03-25T19:22:35.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.646+0000 7f06a413d640 10 monclient: build_initial_monmap 2024-03-25T19:22:35.647 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.646+0000 7f06a413d640 10 monclient: monmap: 2024-03-25T19:22:35.647 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:35.647 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.647 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:22:35.648323+0000 2024-03-25T19:22:35.647 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:22:35.648323+0000 2024-03-25T19:22:35.647 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:35.647 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.648 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:35.648 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:35.648 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:35.648 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.646+0000 7f06a413d640 1 Processor -- start 2024-03-25T19:22:35.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 1 -- start start 2024-03-25T19:22:35.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 10 monclient: init 2024-03-25T19:22:35.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:35.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:35.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 10 monclient(hunting): picked mon.noname-c con 0x7f069c0b4700 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:35.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 10 monclient(hunting): picked mon.noname-b con 0x7f069c142a20 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:35.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 10 monclient(hunting): picked mon.noname-a con 0x7f069c0a96b0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:35.649 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0a9d60 con 0x7f069c0a96b0 2024-03-25T19:22:35.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c13ff10 con 0x7f069c0b4700 2024-03-25T19:22:35.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0aa1f0 con 0x7f069c142a20 2024-03-25T19:22:35.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:35.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.647+0000 7f06a413d640 10 monclient(hunting): authenticate will time out at 5586.932320s 2024-03-25T19:22:35.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.648+0000 7f06a26da640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f069c0a96b0 0x7f069c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.151:36358/0 (socket says 172.21.15.151:36358) 2024-03-25T19:22:35.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.648+0000 7f06a26da640 1 -- 172.21.15.151:0/3081027822 learned_addr learned my addr 172.21.15.151:0/3081027822 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:35.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.650+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986538595 0 0) 0x7f069c0aa1f0 con 0x7f069c142a20 2024-03-25T19:22:35.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.650+0000 7f06a0ed7640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.650+0000 7f06a0ed7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.650+0000 7f06a0ed7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.650+0000 7f06a0ed7640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:22:35.651 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.650+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0688003520 con 0x7f069c142a20 2024-03-25T19:22:35.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.650+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408111203 0 0) 0x7f069c0a9d60 con 0x7f069c0a96b0 2024-03-25T19:22:35.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:22:35.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c0aa1f0 con 0x7f069c0a96b0 2024-03-25T19:22:35.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987979303 0 0) 0x7f069c13ff10 con 0x7f069c0b4700 2024-03-25T19:22:35.652 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:22:35.653 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.651+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c0a9d60 con 0x7f069c0b4700 2024-03-25T19:22:35.654 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3803163164 0 0) 0x7f0688003520 con 0x7f069c142a20 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 >> v1:172.21.15.151:6789/0 conn(0x7f069c0b4700 legacy=0x7f069c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 >> v1:172.21.15.123:6789/0 conn(0x7f069c0a96b0 legacy=0x7f069c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 10 monclient: _finish_hunting 0 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 1 monclient: found mon.noname-b 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c0b4ae0 con 0x7f069c142a20 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 10 monclient: _finish_auth 0 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 10 monclient: _check_auth_tickets 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0698001690 con 0x7f069c142a20 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a413d640 5 monclient: authenticate success, global_id 4130 2024-03-25T19:22:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a413d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a0ed7640 10 monclient: dump: 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.656 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.653+0000 7f06a413d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.680+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06980017f0 con 0x7f069c142a20 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.680+0000 7f06a0ed7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.680+0000 7f06a0ed7640 1 -- 172.21.15.151:0/3081027822 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0698001c00 con 0x7f069c142a20 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.680+0000 7f06a0ed7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a0ed7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a0ed7640 10 monclient: dump: 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.682 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.683 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.683 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a413d640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:35.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a413d640 4 set_mon_vals no callback set 2024-03-25T19:22:35.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a413d640 10 monclient: shutdown 2024-03-25T19:22:35.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a413d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:35.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a413d640 1 -- 172.21.15.151:0/3081027822 >> v1:172.21.15.123:6790/0 conn(0x7f069c142a20 legacy=0x7f069c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a413d640 1 -- 172.21.15.151:0/3081027822 shutdown_connections 2024-03-25T19:22:35.685 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a413d640 1 -- 172.21.15.151:0/3081027822 >> 172.21.15.151:0/3081027822 conn(0x7f069c0668e0 msgr2=0x7f069c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a413d640 1 -- 172.21.15.151:0/3081027822 shutdown_connections 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.681+0000 7f06a413d640 1 -- 172.21.15.151:0/3081027822 wait complete. 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.682+0000 7f06a413d640 10 monclient: build_initial_monmap 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.682+0000 7f06a413d640 10 monclient: monmap: 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:35.686 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.682+0000 7f06a413d640 1 librados: starting msgr at 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.682+0000 7f06a413d640 1 librados: starting objecter 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.682+0000 7f06a413d640 1 Processor -- start 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.682+0000 7f06a413d640 1 -- start start 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.682+0000 7f06a413d640 1 librados: setting wanted keys 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.682+0000 7f06a413d640 1 librados: calling monclient init 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.682+0000 7f06a413d640 10 monclient: init 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a413d640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a413d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a413d640 10 monclient(hunting): picked mon.noname-b con 0x7f069c0a96b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a413d640 10 monclient(hunting): picked mon.noname-c con 0x7f069c0b4700 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a413d640 10 monclient(hunting): picked mon.noname-a con 0x7f069c142a20 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a413d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c133a00 con 0x7f069c142a20 2024-03-25T19:22:35.687 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a413d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c13c5a0 con 0x7f069c0a96b0 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a413d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0ac6a0 con 0x7f069c0b4700 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a413d640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a413d640 10 monclient(hunting): authenticate will time out at 5586.968007s 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a26da640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f069c142a20 0x7f069c1332f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.151:36372/0 (socket says 172.21.15.151:36372) 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.683+0000 7f06a26da640 1 -- 172.21.15.151:0/3913816740 learned_addr learned my addr 172.21.15.151:0/3913816740 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.684+0000 7f06927fc640 10 client.?.objecter ms_handle_connect 0x7f069c142a20 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.684+0000 7f06927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.684+0000 7f06927fc640 10 client.?.objecter ms_handle_connect 0x7f069c0b4700 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.684+0000 7f06927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.684+0000 7f06927fc640 10 client.?.objecter ms_handle_connect 0x7f069c0a96b0 2024-03-25T19:22:35.688 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.684+0000 7f06927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:35.689 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:35.684+0000 7f866c810040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:22:35.689 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718585209 0 0) 0x7f069c0ac6a0 con 0x7f069c0b4700 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0680004eb0 con 0x7f069c0b4700 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139290272 0 0) 0x7f069c133a00 con 0x7f069c142a20 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.686+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c0ac6a0 con 0x7f069c142a20 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.687+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291601272 0 0) 0x7f069c13c5a0 con 0x7f069c0a96b0 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.687+0000 7f06927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.687+0000 7f06927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:35.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.687+0000 7f06927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:35.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.687+0000 7f06927fc640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:22:35.691 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.687+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c133a00 con 0x7f069c0a96b0 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2646640602 0 0) 0x7f0680004eb0 con 0x7f069c0b4700 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f069c13c5a0 con 0x7f069c0b4700 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0698001b00 con 0x7f069c0b4700 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 10 monclient(hunting): dump: 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.692 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.693 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.693 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.693 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.693 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:35.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1740648976 0 0) 0x7f069c0ac6a0 con 0x7f069c142a20 2024-03-25T19:22:35.693 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0680004eb0 con 0x7f069c142a20 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0694001c50 con 0x7f069c142a20 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.691+0000 7f06927fc640 10 monclient(hunting): dump: 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.694 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.695 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.695 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.695 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2174934636 0 0) 0x7f069c133a00 con 0x7f069c0a96b0 2024-03-25T19:22:35.696 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f069c0ac6a0 con 0x7f069c0a96b0 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f068c001830 con 0x7f069c0a96b0 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 10 monclient(hunting): dump: 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2698893732 0 0) 0x7f069c13c5a0 con 0x7f069c0b4700 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 >> v1:172.21.15.151:6789/0 conn(0x7f069c0a96b0 legacy=0x7f069c138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 >> v1:172.21.15.123:6789/0 conn(0x7f069c142a20 legacy=0x7f069c1332f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 1 monclient: found mon.c 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c13caa0 con 0x7f069c0b4700 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06a413d640 5 monclient: authenticate success, global_id 4133 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06a413d640 10 monclient: _renew_subs 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06a413d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06a413d640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c139330 con 0x7f069c0b4700 2024-03-25T19:22:35.698 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06a413d640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:35.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06a413d640 10 monclient: _renew_subs 2024-03-25T19:22:35.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06a413d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:22:35.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06a413d640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c13b7e0 con 0x7f069c0b4700 2024-03-25T19:22:35.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.693+0000 7f06a413d640 1 librados: init done 2024-03-25T19:22:35.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.694+0000 7f06a413d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:35.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.694+0000 7f06a413d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:35.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.694+0000 7f06a413d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:22:35.699 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.694+0000 7f06a413d640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c13ba90 con 0x7f069c0b4700 2024-03-25T19:22:35.735 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:35.735+0000 7f866c810040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:22:35.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.888+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06980038a0 con 0x7f069c0b4700 2024-03-25T19:22:35.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.888+0000 7f06927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:35.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.888+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0698003a10 con 0x7f069c0b4700 2024-03-25T19:22:35.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.888+0000 7f06927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:35.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.888+0000 7f06927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:35.889 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.888+0000 7f06927fc640 10 monclient: dump: 2024-03-25T19:22:35.889 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:35.889 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:35.889 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.890 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:35.890 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:35.890 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:35.890 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:35.890 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:35.890 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:35.890 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:35.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.888+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0698003f00 con 0x7f069c0b4700 2024-03-25T19:22:35.890 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.888+0000 7f0693fff640 4 set_mon_vals no callback set 2024-03-25T19:22:35.891 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.890+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3436693575 0 0) 0x7f0698005340 con 0x7f069c0b4700 2024-03-25T19:22:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.890+0000 7f06927fc640 10 client.4133.objecter ms_dispatch 0x7f069c000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:22:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.890+0000 7f06927fc640 3 client.4133.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:22:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.890+0000 7f06927fc640 3 client.4133.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:22:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.890+0000 7f06927fc640 20 client.4133.objecter dump_active .. 0 homeless 2024-03-25T19:22:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.890+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0698004f20 con 0x7f069c0b4700 2024-03-25T19:22:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.891+0000 7f06927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.891+0000 7f06927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:35.900 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:35.900+0000 7f327d637040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:22:35.965 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.965+0000 7f06a413d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "96cf3100-b688-4a78-87ef-06b8a6806ac3", "id": 2}] 2024-03-25T19:22:36.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.965+0000 7f06a413d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "96cf3100-b688-4a78-87ef-06b8a6806ac3", "id": 2}] 2024-03-25T19:22:36.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.965+0000 7f06a413d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:22:36.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:35.965+0000 7f06a413d640 1 -- 172.21.15.151:0/3913816740 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd new", "uuid": "96cf3100-b688-4a78-87ef-06b8a6806ac3", "id": 2} v 0) v1 -- 0x7f069c13c730 con 0x7f069c0b4700 2024-03-25T19:22:36.038 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:36.037+0000 7f327d637040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:22:36.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.090+0000 7f06927fc640 1 -- 172.21.15.151:0/3913816740 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "96cf3100-b688-4a78-87ef-06b8a6806ac3", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3109439303 0 216599125) 0x7f069c13c730 con 0x7f069c0b4700 2024-03-25T19:22:36.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.090+0000 7f06927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "96cf3100-b688-4a78-87ef-06b8a6806ac3", "id": 2}] 2024-03-25T19:22:36.091 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.090+0000 7f06927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:36.091 INFO:teuthology.orchestra.run.smithi151.stdout:2 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 10 librados: watch_flush enter 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 10 librados: watch_flush exit 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 20 client.4133.objecter shutdown clearing up homeless session... 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 10 client.4133.objecter successfully canceled tick 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 10 monclient: shutdown 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 1 -- 172.21.15.151:0/3913816740 >> v1:172.21.15.123:6790/0 conn(0x7f069c0b4700 legacy=0x7f069c132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 1 -- 172.21.15.151:0/3913816740 shutdown_connections 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 1 -- 172.21.15.151:0/3913816740 >> 172.21.15.151:0/3913816740 conn(0x7f069c0668e0 msgr2=0x7f069c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 1 -- 172.21.15.151:0/3913816740 shutdown_connections 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 1 -- 172.21.15.151:0/3913816740 wait complete. 2024-03-25T19:22:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.091+0000 7f06a413d640 1 librados: shutdown 2024-03-25T19:22:36.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new f0fa10c2-a2a8-4392-9d03-a1b93522a284 3 2024-03-25T19:22:36.138 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:36.138+0000 7f866c810040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:22:36.168 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:36.167+0000 7f327d637040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:22:36.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.200+0000 7f9b370f8640 10 monclient: get_monmap_and_config 2024-03-25T19:22:36.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.200+0000 7f9b370f8640 10 monclient: build_initial_monmap 2024-03-25T19:22:36.201 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.200+0000 7f9b370f8640 10 monclient: monmap: 2024-03-25T19:22:36.201 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:36.201 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.201 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:22:36.202573+0000 2024-03-25T19:22:36.201 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:22:36.202573+0000 2024-03-25T19:22:36.201 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:36.202 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.202 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:36.202 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:36.202 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:36.202 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.201+0000 7f9b370f8640 1 Processor -- start 2024-03-25T19:22:36.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.201+0000 7f9b370f8640 1 -- start start 2024-03-25T19:22:36.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.201+0000 7f9b370f8640 10 monclient: init 2024-03-25T19:22:36.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.201+0000 7f9b370f8640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:36.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.201+0000 7f9b370f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:22:36.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.201+0000 7f9b370f8640 10 monclient(hunting): picked mon.noname-b con 0x7f9b300b4700 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:36.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.201+0000 7f9b370f8640 10 monclient(hunting): picked mon.noname-c con 0x7f9b30142a20 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:36.203 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.201+0000 7f9b370f8640 10 monclient(hunting): picked mon.noname-a con 0x7f9b300a96b0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:36.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.201+0000 7f9b370f8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b300a9d60 con 0x7f9b300a96b0 2024-03-25T19:22:36.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.202+0000 7f9b370f8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3013ff10 con 0x7f9b30142a20 2024-03-25T19:22:36.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.202+0000 7f9b370f8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b300aa1f0 con 0x7f9b300b4700 2024-03-25T19:22:36.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.202+0000 7f9b370f8640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:36.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.202+0000 7f9b370f8640 10 monclient(hunting): authenticate will time out at 5587.486579s 2024-03-25T19:22:36.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.202+0000 7f9b35695640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f9b300a96b0 0x7f9b3013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.151:36382/0 (socket says 172.21.15.151:36382) 2024-03-25T19:22:36.204 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.202+0000 7f9b35695640 1 -- 172.21.15.151:0/194629164 learned_addr learned my addr 172.21.15.151:0/194629164 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676069335 0 0) 0x7f9b300a9d60 con 0x7f9b300a96b0 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b20003520 con 0x7f9b300a96b0 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206555553 0 0) 0x7f9b3013ff10 con 0x7f9b30142a20 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.204+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b300a9d60 con 0x7f9b30142a20 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.205+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026302302 0 0) 0x7f9b300aa1f0 con 0x7f9b300b4700 2024-03-25T19:22:36.206 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.205+0000 7f9b2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.205+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.205+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.205+0000 7f9b2f7fe640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:22:36.207 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.205+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b3013ff10 con 0x7f9b300b4700 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3067632127 0 0) 0x7f9b20003520 con 0x7f9b300a96b0 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 >> v1:172.21.15.123:6790/0 conn(0x7f9b300b4700 legacy=0x7f9b30142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 >> v1:172.21.15.151:6789/0 conn(0x7f9b30142a20 legacy=0x7f9b3013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 1 monclient: found mon.noname-a 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b300b4ae0 con 0x7f9b300a96b0 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b28001a10 con 0x7f9b300a96b0 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b370f8640 5 monclient: authenticate success, global_id 4134 2024-03-25T19:22:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b370f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.207+0000 7f9b2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.208+0000 7f9b2f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.208+0000 7f9b2f7fe640 10 monclient: dump: 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.210 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.208+0000 7f9b370f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:36.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.209+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b28001b70 con 0x7f9b300a96b0 2024-03-25T19:22:36.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.209+0000 7f9b2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:36.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.209+0000 7f9b2f7fe640 1 -- 172.21.15.151:0/194629164 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b28001f80 con 0x7f9b300a96b0 2024-03-25T19:22:36.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.209+0000 7f9b2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.209+0000 7f9b2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:36.212 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.209+0000 7f9b2f7fe640 10 monclient: dump: 2024-03-25T19:22:36.212 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.212 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.212 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.212 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.209+0000 7f9b370f8640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.209+0000 7f9b370f8640 4 set_mon_vals no callback set 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.210+0000 7f9b370f8640 10 monclient: shutdown 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.210+0000 7f9b370f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.210+0000 7f9b370f8640 1 -- 172.21.15.151:0/194629164 >> v1:172.21.15.123:6789/0 conn(0x7f9b300a96b0 legacy=0x7f9b3013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.210+0000 7f9b370f8640 1 -- 172.21.15.151:0/194629164 shutdown_connections 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.210+0000 7f9b370f8640 1 -- 172.21.15.151:0/194629164 >> 172.21.15.151:0/194629164 conn(0x7f9b300668e0 msgr2=0x7f9b30066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.210+0000 7f9b370f8640 1 -- 172.21.15.151:0/194629164 shutdown_connections 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.210+0000 7f9b370f8640 1 -- 172.21.15.151:0/194629164 wait complete. 2024-03-25T19:22:36.213 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.211+0000 7f9b370f8640 10 monclient: build_initial_monmap 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.211+0000 7f9b370f8640 10 monclient: monmap: 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.211+0000 7f9b370f8640 1 librados: starting msgr at 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.211+0000 7f9b370f8640 1 librados: starting objecter 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.211+0000 7f9b370f8640 1 Processor -- start 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.211+0000 7f9b370f8640 1 -- start start 2024-03-25T19:22:36.214 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.211+0000 7f9b370f8640 1 librados: setting wanted keys 2024-03-25T19:22:36.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.211+0000 7f9b370f8640 1 librados: calling monclient init 2024-03-25T19:22:36.215 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.211+0000 7f9b370f8640 10 monclient: init 2024-03-25T19:22:36.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b370f8640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:36.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b370f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:36.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b370f8640 10 monclient(hunting): picked mon.noname-b con 0x7f9b300a96b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:36.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b370f8640 10 monclient(hunting): picked mon.noname-c con 0x7f9b300b4700 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:36.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b370f8640 10 monclient(hunting): picked mon.noname-a con 0x7f9b30142a20 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:36.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b370f8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b300aba20 con 0x7f9b30142a20 2024-03-25T19:22:36.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b370f8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b300abd00 con 0x7f9b300a96b0 2024-03-25T19:22:36.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b370f8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3013bd80 con 0x7f9b300b4700 2024-03-25T19:22:36.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b370f8640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:36.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b370f8640 10 monclient(hunting): authenticate will time out at 5587.496782s 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b35695640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f9b30142a20 0x7f9b301e2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.151:36394/0 (socket says 172.21.15.151:36394) 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b35695640 1 -- 172.21.15.151:0/2995188553 learned_addr learned my addr 172.21.15.151:0/2995188553 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b2d7fa640 10 client.?.objecter ms_handle_connect 0x7f9b30142a20 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.212+0000 7f9b2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.213+0000 7f9b2d7fa640 10 client.?.objecter ms_handle_connect 0x7f9b300b4700 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.213+0000 7f9b2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.213+0000 7f9b2d7fa640 10 client.?.objecter ms_handle_connect 0x7f9b300a96b0 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.213+0000 7f9b2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.214+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212104986 0 0) 0x7f9b300aba20 con 0x7f9b30142a20 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.214+0000 7f9b2d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.214+0000 7f9b2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.214+0000 7f9b2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.214+0000 7f9b2d7fa640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.214+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b14004eb0 con 0x7f9b30142a20 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483357735 0 0) 0x7f9b300abd00 con 0x7f9b300a96b0 2024-03-25T19:22:36.218 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:22:36.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b300aba20 con 0x7f9b300a96b0 2024-03-25T19:22:36.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302700446 0 0) 0x7f9b3013bd80 con 0x7f9b300b4700 2024-03-25T19:22:36.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:22:36.219 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.215+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b300abd00 con 0x7f9b300b4700 2024-03-25T19:22:36.219 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:36.216+0000 7f327d637040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:22:36.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.219+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2557275719 0 0) 0x7f9b14004eb0 con 0x7f9b30142a20 2024-03-25T19:22:36.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.219+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b3013bd80 con 0x7f9b30142a20 2024-03-25T19:22:36.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.219+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b28001ec0 con 0x7f9b30142a20 2024-03-25T19:22:36.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.219+0000 7f9b2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.219+0000 7f9b2d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:36.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.219+0000 7f9b2d7fa640 10 monclient(hunting): dump: 2024-03-25T19:22:36.220 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.220 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.220 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.220 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.221 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.221 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.221 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.221 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.221 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.221 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.221 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.219+0000 7f9b2d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.220+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616347006 0 0) 0x7f9b300aba20 con 0x7f9b300a96b0 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.220+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b14004eb0 con 0x7f9b300a96b0 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.220+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b24001870 con 0x7f9b300a96b0 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.220+0000 7f9b2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.220+0000 7f9b2d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.220+0000 7f9b2d7fa640 10 monclient(hunting): dump: 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.222 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.223 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.223 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.223 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.223 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.223 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.222+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3253851857 0 0) 0x7f9b300abd00 con 0x7f9b300b4700 2024-03-25T19:22:36.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.222+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b300aba20 con 0x7f9b300b4700 2024-03-25T19:22:36.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.222+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b1c0018f0 con 0x7f9b300b4700 2024-03-25T19:22:36.223 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.222+0000 7f9b2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.222+0000 7f9b2d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.222+0000 7f9b2d7fa640 10 monclient(hunting): dump: 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.224 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 688572306 0 0) 0x7f9b14004eb0 con 0x7f9b300a96b0 2024-03-25T19:22:36.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 >> v1:172.21.15.123:6790/0 conn(0x7f9b300b4700 legacy=0x7f9b301392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 >> v1:172.21.15.123:6789/0 conn(0x7f9b30142a20 legacy=0x7f9b301e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:22:36.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 1 monclient: found mon.b 2024-03-25T19:22:36.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:36.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:36.226 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3013ccb0 con 0x7f9b300a96b0 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b370f8640 5 monclient: authenticate success, global_id 4138 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b370f8640 10 monclient: _renew_subs 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b370f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b370f8640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3013ac10 con 0x7f9b300a96b0 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b370f8640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b370f8640 10 monclient: _renew_subs 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b370f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b370f8640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3013b8d0 con 0x7f9b300a96b0 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.223+0000 7f9b370f8640 1 librados: init done 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.224+0000 7f9b370f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.224+0000 7f9b370f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:36.227 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.224+0000 7f9b370f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:36.228 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.224+0000 7f9b370f8640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3013bf10 con 0x7f9b300a96b0 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.248+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b24003610 con 0x7f9b300a96b0 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.248+0000 7f9b2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.248+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b24001d80 con 0x7f9b300a96b0 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.248+0000 7f9b2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.248+0000 7f9b2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.248+0000 7f9b2d7fa640 10 monclient: dump: 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.248+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9b24003d30 con 0x7f9b300a96b0 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.248+0000 7f9b2effd640 4 set_mon_vals no callback set 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.249+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1687770683 0 0) 0x7f9b24005440 con 0x7f9b300a96b0 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.249+0000 7f9b2d7fa640 10 client.4138.objecter ms_dispatch 0x7f9b30000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.249+0000 7f9b2d7fa640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.249+0000 7f9b2d7fa640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:22:36.251 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.249+0000 7f9b2d7fa640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T19:22:36.277 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:36.278+0000 7f866c810040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:22:36.348 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:36.347+0000 7f327d637040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:22:36.402 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:36.402+0000 7f866c810040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:22:36.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.428+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9b240050c0 con 0x7f9b300a96b0 2024-03-25T19:22:36.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.429+0000 7f9b2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:36.429 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.429+0000 7f9b2d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:36.446 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:36.445+0000 7f327d637040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:22:36.449 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:36.449+0000 7f866c810040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:22:36.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.501+0000 7f9b370f8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f0fa10c2-a2a8-4392-9d03-a1b93522a284", "id": 3}] 2024-03-25T19:22:36.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.501+0000 7f9b370f8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f0fa10c2-a2a8-4392-9d03-a1b93522a284", "id": 3}] 2024-03-25T19:22:36.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.501+0000 7f9b370f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:36.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.501+0000 7f9b370f8640 1 -- 172.21.15.151:0/2995188553 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f0fa10c2-a2a8-4392-9d03-a1b93522a284", "id": 3} v 0) v1 -- 0x7f9b3013bb80 con 0x7f9b300a96b0 2024-03-25T19:22:36.564 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:36.564+0000 7f866c810040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:22:36.605 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:36.604+0000 7f327d637040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:22:36.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.623+0000 7f9b2d7fa640 1 -- 172.21.15.151:0/2995188553 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f0fa10c2-a2a8-4392-9d03-a1b93522a284", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2468746042 0 525046306) 0x7f9b24005650 con 0x7f9b300a96b0 2024-03-25T19:22:36.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.623+0000 7f9b2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f0fa10c2-a2a8-4392-9d03-a1b93522a284", "id": 3}] 2024-03-25T19:22:36.624 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.623+0000 7f9b2d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:36.624 INFO:teuthology.orchestra.run.smithi151.stdout:3 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.624+0000 7f9b370f8640 10 librados: watch_flush enter 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.624+0000 7f9b370f8640 10 librados: watch_flush exit 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.624+0000 7f9b370f8640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.624+0000 7f9b370f8640 10 client.4138.objecter successfully canceled tick 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.624+0000 7f9b370f8640 10 monclient: shutdown 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.624+0000 7f9b370f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.624+0000 7f9b370f8640 1 -- 172.21.15.151:0/2995188553 >> v1:172.21.15.151:6789/0 conn(0x7f9b300a96b0 legacy=0x7f9b300ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.624+0000 7f9b370f8640 1 -- 172.21.15.151:0/2995188553 shutdown_connections 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.624+0000 7f9b370f8640 1 -- 172.21.15.151:0/2995188553 >> 172.21.15.151:0/2995188553 conn(0x7f9b300668e0 msgr2=0x7f9b3013d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.625+0000 7f9b370f8640 1 -- 172.21.15.151:0/2995188553 shutdown_connections 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.625+0000 7f9b370f8640 1 -- 172.21.15.151:0/2995188553 wait complete. 2024-03-25T19:22:36.626 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.625+0000 7f9b370f8640 1 librados: shutdown 2024-03-25T19:22:36.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 1738fb73-735d-4c91-9442-f6f6aa9c3a05 4 2024-03-25T19:22:36.650 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:36.649+0000 7f327d637040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:22:36.665 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:36.666+0000 7f866c810040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.735+0000 7f018de8d640 10 monclient: get_monmap_and_config 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.735+0000 7f018de8d640 10 monclient: build_initial_monmap 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.735+0000 7f018de8d640 10 monclient: monmap: 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:22:36.736958+0000 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:22:36.736958+0000 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:36.736 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.735+0000 7f018de8d640 1 Processor -- start 2024-03-25T19:22:36.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.735+0000 7f018de8d640 1 -- start start 2024-03-25T19:22:36.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.735+0000 7f018de8d640 10 monclient: init 2024-03-25T19:22:36.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.736+0000 7f018de8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:36.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.736+0000 7f018de8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:36.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.736+0000 7f018de8d640 10 monclient(hunting): picked mon.noname-a con 0x7f01880b4700 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:36.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.736+0000 7f018de8d640 10 monclient(hunting): picked mon.noname-c con 0x7f0188142a20 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:36.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.736+0000 7f018de8d640 10 monclient(hunting): picked mon.noname-b con 0x7f01880a96b0 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:36.738 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.736+0000 7f018de8d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01880a9d60 con 0x7f01880b4700 2024-03-25T19:22:36.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.736+0000 7f018de8d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018813ff10 con 0x7f0188142a20 2024-03-25T19:22:36.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.736+0000 7f018de8d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01880aa1f0 con 0x7f01880a96b0 2024-03-25T19:22:36.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.736+0000 7f018de8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:36.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.736+0000 7f018de8d640 10 monclient(hunting): authenticate will time out at 5588.021017s 2024-03-25T19:22:36.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.737+0000 7f01877fe640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f01880b4700 0x7f0188142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.151:36410/0 (socket says 172.21.15.151:36410) 2024-03-25T19:22:36.739 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.737+0000 7f01877fe640 1 -- 172.21.15.151:0/3076572217 learned_addr learned my addr 172.21.15.151:0/3076572217 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:36.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239294018 0 0) 0x7f01880a9d60 con 0x7f01880b4700 2024-03-25T19:22:36.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:22:36.740 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0174003520 con 0x7f01880b4700 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364717389 0 0) 0x7f018813ff10 con 0x7f0188142a20 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01880a9d60 con 0x7f0188142a20 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184599541 0 0) 0x7f01880aa1f0 con 0x7f01880a96b0 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:22:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.739+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f018813ff10 con 0x7f01880a96b0 2024-03-25T19:22:36.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531868648 0 0) 0x7f0174003520 con 0x7f01880b4700 2024-03-25T19:22:36.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 >> v1:172.21.15.123:6790/0 conn(0x7f01880a96b0 legacy=0x7f018813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 >> v1:172.21.15.151:6789/0 conn(0x7f0188142a20 legacy=0x7f018813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:36.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 1 monclient: found mon.noname-a 2024-03-25T19:22:36.743 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01880b4ae0 con 0x7f01880b4700 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0178001690 con 0x7f01880b4700 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f0186ffd640 10 monclient: dump: 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.744 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.745 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.745 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.745 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.745 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f018de8d640 5 monclient: authenticate success, global_id 4140 2024-03-25T19:22:36.745 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.742+0000 7f018de8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:36.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01780017f0 con 0x7f01880b4700 2024-03-25T19:22:36.746 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f0186ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f0186ffd640 1 -- 172.21.15.151:0/3076572217 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0178001c00 con 0x7f01880b4700 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f018de8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f018de8d640 4 set_mon_vals no callback set 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f0186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f0186ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f0186ffd640 10 monclient: dump: 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f018de8d640 10 monclient: shutdown 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f018de8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.744+0000 7f018de8d640 1 -- 172.21.15.151:0/3076572217 >> v1:172.21.15.123:6789/0 conn(0x7f01880b4700 legacy=0x7f0188142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.745+0000 7f018de8d640 1 -- 172.21.15.151:0/3076572217 shutdown_connections 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.745+0000 7f018de8d640 1 -- 172.21.15.151:0/3076572217 >> 172.21.15.151:0/3076572217 conn(0x7f01880668e0 msgr2=0x7f0188066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.745+0000 7f018de8d640 1 -- 172.21.15.151:0/3076572217 shutdown_connections 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.745+0000 7f018de8d640 1 -- 172.21.15.151:0/3076572217 wait complete. 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.745+0000 7f018de8d640 10 monclient: build_initial_monmap 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.745+0000 7f018de8d640 10 monclient: monmap: 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2024-03-25T19:22:36.748 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.745+0000 7f018de8d640 1 librados: starting msgr at 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.745+0000 7f018de8d640 1 librados: starting objecter 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 1 Processor -- start 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 1 -- start start 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 1 librados: setting wanted keys 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 1 librados: calling monclient init 2024-03-25T19:22:36.749 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 10 monclient: init 2024-03-25T19:22:36.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:36.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 10 monclient(hunting): picked mon.noname-a con 0x7f01880a96b0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 10 monclient(hunting): picked mon.noname-b con 0x7f01880b4700 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 10 monclient(hunting): picked mon.noname-c con 0x7f0188142a20 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01880aba20 con 0x7f01880a96b0 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01880ad830 con 0x7f01880b4700 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01881db6a0 con 0x7f0188142a20 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.746+0000 7f018de8d640 10 monclient(hunting): authenticate will time out at 5588.031430s 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.747+0000 7f017effd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f01880b4700 0x7f01881da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37540/0 (socket says 172.21.15.151:37540) 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.747+0000 7f017effd640 1 -- 172.21.15.151:0/2085022245 learned_addr learned my addr 172.21.15.151:0/2085022245 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.747+0000 7f0184ff9640 10 client.?.objecter ms_handle_connect 0x7f01880b4700 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.747+0000 7f0184ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.748+0000 7f0184ff9640 10 client.?.objecter ms_handle_connect 0x7f01880a96b0 2024-03-25T19:22:36.752 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.748+0000 7f0184ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.749+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418548577 0 0) 0x7f01880ad830 con 0x7f01880b4700 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.749+0000 7f0184ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.749+0000 7f0184ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.749+0000 7f0184ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.749+0000 7f0184ff9640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.749+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0168004eb0 con 0x7f01880b4700 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.750+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327810860 0 0) 0x7f01880aba20 con 0x7f01880a96b0 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.750+0000 7f0184ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.750+0000 7f0184ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.750+0000 7f0184ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.750+0000 7f0184ff9640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.750+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01880ad830 con 0x7f01880a96b0 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.750+0000 7f0184ff9640 10 client.?.objecter ms_handle_connect 0x7f0188142a20 2024-03-25T19:22:36.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.750+0000 7f0184ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:36.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.752+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651556595 0 0) 0x7f01881db6a0 con 0x7f0188142a20 2024-03-25T19:22:36.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.752+0000 7f0184ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:36.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.752+0000 7f0184ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:36.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.752+0000 7f0184ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:36.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.752+0000 7f0184ff9640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:22:36.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.752+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01880aba20 con 0x7f0188142a20 2024-03-25T19:22:36.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.754+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 929751558 0 0) 0x7f0168004eb0 con 0x7f01880b4700 2024-03-25T19:22:36.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.754+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01881db6a0 con 0x7f01880b4700 2024-03-25T19:22:36.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.754+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0170001a30 con 0x7f01880b4700 2024-03-25T19:22:36.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.754+0000 7f0184ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.754+0000 7f0184ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:36.755 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.754+0000 7f0184ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:36.755 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.754+0000 7f0184ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.755+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1799739474 0 0) 0x7f01880ad830 con 0x7f01880a96b0 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.755+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0168004eb0 con 0x7f01880a96b0 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.755+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0178001b00 con 0x7f01880a96b0 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.755+0000 7f0184ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.755+0000 7f0184ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.755+0000 7f0184ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.757 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.758 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.758 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.758 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 481661632 0 0) 0x7f01881db6a0 con 0x7f01880b4700 2024-03-25T19:22:36.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 >> v1:172.21.15.123:6790/0 conn(0x7f0188142a20 legacy=0x7f01881daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 >> v1:172.21.15.123:6789/0 conn(0x7f01880a96b0 legacy=0x7f01881d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:36.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:22:36.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 1 monclient: found mon.b 2024-03-25T19:22:36.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:36.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:36.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01881d4e90 con 0x7f01880b4700 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 10 monclient: _finish_auth 0 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f0184ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f018de8d640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f018de8d640 10 monclient: _renew_subs 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f018de8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f018de8d640 1 -- 172.21.15.151:0/2085022245 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018813ba80 con 0x7f01880b4700 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f018de8d640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f018de8d640 10 monclient: _renew_subs 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f018de8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f018de8d640 1 -- 172.21.15.151:0/2085022245 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01880ac790 con 0x7f01880b4700 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.757+0000 7f018de8d640 1 librados: init done 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.758+0000 7f018de8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.758+0000 7f018de8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:36.760 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.758+0000 7f018de8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:36.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.758+0000 7f018de8d640 1 -- 172.21.15.151:0/2085022245 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01880ab880 con 0x7f01880b4700 2024-03-25T19:22:36.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0170001c70 con 0x7f01880b4700 2024-03-25T19:22:36.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:36.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f01700038c0 con 0x7f01880b4700 2024-03-25T19:22:36.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:36.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 10 monclient: dump: 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0170003e30 con 0x7f01880b4700 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f01867fc640 4 set_mon_vals no callback set 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3539357628 0 0) 0x7f0170006670 con 0x7f01880b4700 2024-03-25T19:22:36.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 10 client.4144.objecter ms_dispatch 0x7f0188000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:22:36.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:22:36.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:22:36.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.761+0000 7f0184ff9640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T19:22:36.813 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:36.812+0000 7f327d637040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:22:36.826 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:36.826+0000 7f866c810040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:22:36.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.856+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0170006270 con 0x7f01880b4700 2024-03-25T19:22:36.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.856+0000 7f0184ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:36.857 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.856+0000 7f0184ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:36.868 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:36.868+0000 7f866c810040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:22:36.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.928+0000 7f018de8d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1738fb73-735d-4c91-9442-f6f6aa9c3a05", "id": 4}] 2024-03-25T19:22:36.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.929+0000 7f018de8d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1738fb73-735d-4c91-9442-f6f6aa9c3a05", "id": 4}] 2024-03-25T19:22:36.929 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.929+0000 7f018de8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:36.930 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:36.929+0000 7f018de8d640 1 -- 172.21.15.151:0/2085022245 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1738fb73-735d-4c91-9442-f6f6aa9c3a05", "id": 4} v 0) v1 -- 0x7f018804f730 con 0x7f01880b4700 2024-03-25T19:22:37.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.001+0000 7f0184ff9640 1 -- 172.21.15.151:0/2085022245 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1738fb73-735d-4c91-9442-f6f6aa9c3a05", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 15577078 0 1697012583) 0x7f01700040e0 con 0x7f01880b4700 2024-03-25T19:22:37.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.001+0000 7f0184ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1738fb73-735d-4c91-9442-f6f6aa9c3a05", "id": 4}] 2024-03-25T19:22:37.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.001+0000 7f0184ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:37.002 INFO:teuthology.orchestra.run.smithi151.stdout:4 2024-03-25T19:22:37.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 10 librados: watch_flush enter 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 10 librados: watch_flush exit 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 10 client.4144.objecter successfully canceled tick 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 10 monclient: shutdown 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 1 -- 172.21.15.151:0/2085022245 >> v1:172.21.15.151:6789/0 conn(0x7f01880b4700 legacy=0x7f01881da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 1 -- 172.21.15.151:0/2085022245 shutdown_connections 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 1 -- 172.21.15.151:0/2085022245 >> 172.21.15.151:0/2085022245 conn(0x7f01880668e0 msgr2=0x7f01880b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 1 -- 172.21.15.151:0/2085022245 shutdown_connections 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 1 -- 172.21.15.151:0/2085022245 wait complete. 2024-03-25T19:22:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.002+0000 7f018de8d640 1 librados: shutdown 2024-03-25T19:22:37.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 27d55d50-50bb-48a2-8870-d038cb804e88 5 2024-03-25T19:22:37.033 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:37.034+0000 7f866c810040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:22:37.153 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.153+0000 7ff3a6233640 10 monclient: get_monmap_and_config 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.153+0000 7ff3a6233640 10 monclient: build_initial_monmap 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.153+0000 7ff3a6233640 10 monclient: monmap: 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:22:37.155011+0000 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:22:37.155011+0000 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:37.154 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.153+0000 7ff3a6233640 1 Processor -- start 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.153+0000 7ff3a6233640 1 -- start start 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.153+0000 7ff3a6233640 10 monclient: init 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.154+0000 7ff3a6233640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.154+0000 7ff3a6233640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.154+0000 7ff3a6233640 10 monclient(hunting): picked mon.noname-c con 0x7ff3a00a9920 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.154+0000 7ff3a6233640 10 monclient(hunting): picked mon.noname-b con 0x7ff3a00b45a0 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.154+0000 7ff3a6233640 10 monclient(hunting): picked mon.noname-a con 0x7ff3a0146450 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.154+0000 7ff3a6233640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a006b100 con 0x7ff3a0146450 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.154+0000 7ff3a6233640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a00b2940 con 0x7ff3a00a9920 2024-03-25T19:22:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.154+0000 7ff3a6233640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a013fec0 con 0x7ff3a00b45a0 2024-03-25T19:22:37.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.154+0000 7ff3a6233640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:37.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.154+0000 7ff3a6233640 10 monclient(hunting): authenticate will time out at 5588.439024s 2024-03-25T19:22:37.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.155+0000 7ff39f7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ff3a00a9920 0x7ff3a013f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37554/0 (socket says 172.21.15.151:37554) 2024-03-25T19:22:37.157 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.155+0000 7ff39f7fe640 1 -- 172.21.15.151:0/2409466472 learned_addr learned my addr 172.21.15.151:0/2409466472 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763326538 0 0) 0x7ff3a00b2940 con 0x7ff3a00a9920 2024-03-25T19:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff388003520 con 0x7ff3a00a9920 2024-03-25T19:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328581456 0 0) 0x7ff3a006b100 con 0x7ff3a0146450 2024-03-25T19:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:22:37.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a00b2940 con 0x7ff3a0146450 2024-03-25T19:22:37.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084650465 0 0) 0x7ff3a013fec0 con 0x7ff3a00b45a0 2024-03-25T19:22:37.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:22:37.159 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.157+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a006b100 con 0x7ff3a00b45a0 2024-03-25T19:22:37.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3796458728 0 0) 0x7ff388003520 con 0x7ff3a00a9920 2024-03-25T19:22:37.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 >> v1:172.21.15.123:6790/0 conn(0x7ff3a00b45a0 legacy=0x7ff3a0145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 >> v1:172.21.15.123:6789/0 conn(0x7ff3a0146450 legacy=0x7ff3a014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.161 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a0140050 con 0x7ff3a00a9920 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff394001690 con 0x7ff3a00a9920 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff3a6233640 5 monclient: authenticate success, global_id 4147 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff3a6233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff39e7fc640 10 monclient: dump: 2024-03-25T19:22:37.162 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.163 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.160+0000 7ff3a6233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:37.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3940017f0 con 0x7ff3a00a9920 2024-03-25T19:22:37.165 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff39e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff39e7fc640 1 -- 172.21.15.151:0/2409466472 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff394001a60 con 0x7ff3a00a9920 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff39e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff39e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff39e7fc640 10 monclient: dump: 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.166 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff3a6233640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff3a6233640 4 set_mon_vals no callback set 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff3a6233640 10 monclient: shutdown 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff3a6233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff3a6233640 1 -- 172.21.15.151:0/2409466472 >> v1:172.21.15.151:6789/0 conn(0x7ff3a00a9920 legacy=0x7ff3a013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff3a6233640 1 -- 172.21.15.151:0/2409466472 shutdown_connections 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff3a6233640 1 -- 172.21.15.151:0/2409466472 >> 172.21.15.151:0/2409466472 conn(0x7ff3a00b7f40 msgr2=0x7ff3a00b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff3a6233640 1 -- 172.21.15.151:0/2409466472 shutdown_connections 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.162+0000 7ff3a6233640 1 -- 172.21.15.151:0/2409466472 wait complete. 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.163+0000 7ff3a6233640 10 monclient: build_initial_monmap 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.163+0000 7ff3a6233640 10 monclient: monmap: 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2024-03-25T19:22:37.167 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.163+0000 7ff3a6233640 1 librados: starting msgr at 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.163+0000 7ff3a6233640 1 librados: starting objecter 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 1 Processor -- start 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 1 -- start start 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 1 librados: setting wanted keys 2024-03-25T19:22:37.168 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 1 librados: calling monclient init 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 10 monclient: init 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 10 monclient(hunting): picked mon.noname-b con 0x7ff3a00a9920 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 10 monclient(hunting): picked mon.noname-a con 0x7ff3a00b45a0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 10 monclient(hunting): picked mon.noname-c con 0x7ff3a0146450 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a00ac0c0 con 0x7ff3a00b45a0 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a00ac330 con 0x7ff3a00a9920 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a01db220 con 0x7ff3a0146450 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.164+0000 7ff3a6233640 10 monclient(hunting): authenticate will time out at 5588.449445s 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.165+0000 7ff39f7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ff3a00a9920 0x7ff3a01da830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37568/0 (socket says 172.21.15.151:37568) 2024-03-25T19:22:37.169 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.165+0000 7ff39f7fe640 1 -- 172.21.15.151:0/2968851836 learned_addr learned my addr 172.21.15.151:0/2968851836 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:37.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.165+0000 7ff387fff640 10 client.?.objecter ms_handle_connect 0x7ff3a00a9920 2024-03-25T19:22:37.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.165+0000 7ff387fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.165+0000 7ff387fff640 10 client.?.objecter ms_handle_connect 0x7ff3a00b45a0 2024-03-25T19:22:37.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.165+0000 7ff387fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.165+0000 7ff387fff640 10 client.?.objecter ms_handle_connect 0x7ff3a0146450 2024-03-25T19:22:37.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.165+0000 7ff387fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097120264 0 0) 0x7ff3a00ac330 con 0x7ff3a00a9920 2024-03-25T19:22:37.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.171 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff38c004eb0 con 0x7ff3a00a9920 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722802256 0 0) 0x7ff3a00ac0c0 con 0x7ff3a00b45a0 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.167+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a00ac330 con 0x7ff3a00b45a0 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.168+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653981674 0 0) 0x7ff3a01db220 con 0x7ff3a0146450 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.168+0000 7ff387fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.168+0000 7ff387fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.168+0000 7ff387fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.172 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.168+0000 7ff387fff640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:22:37.173 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.168+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a00ac0c0 con 0x7ff3a0146450 2024-03-25T19:22:37.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 833557632 0 0) 0x7ff38c004eb0 con 0x7ff3a00a9920 2024-03-25T19:22:37.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3a01db220 con 0x7ff3a00a9920 2024-03-25T19:22:37.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff394001b00 con 0x7ff3a00a9920 2024-03-25T19:22:37.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:37.174 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 10 monclient(hunting): dump: 2024-03-25T19:22:37.174 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.174 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.174 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1582708604 0 0) 0x7ff3a00ac330 con 0x7ff3a00b45a0 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff38c004eb0 con 0x7ff3a00b45a0 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff390001870 con 0x7ff3a00b45a0 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.172+0000 7ff387fff640 10 monclient(hunting): dump: 2024-03-25T19:22:37.175 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.173+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2018887837 0 0) 0x7ff3a00ac0c0 con 0x7ff3a0146450 2024-03-25T19:22:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.173+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3a00ac330 con 0x7ff3a0146450 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.174+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff398001bd0 con 0x7ff3a0146450 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.174+0000 7ff387fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.174+0000 7ff387fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.174+0000 7ff387fff640 10 monclient(hunting): dump: 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.178 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1147861985 0 0) 0x7ff38c004eb0 con 0x7ff3a00b45a0 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 >> v1:172.21.15.123:6790/0 conn(0x7ff3a0146450 legacy=0x7ff3a0144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 >> v1:172.21.15.151:6789/0 conn(0x7ff3a00a9920 legacy=0x7ff3a01da830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 10 monclient: _finish_hunting 0 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 1 monclient: found mon.a 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a01daf40 con 0x7ff3a00b45a0 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 10 monclient: _finish_auth 0 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 10 monclient: _check_auth_tickets 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff387fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff3a6233640 5 monclient: authenticate success, global_id 4149 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff3a6233640 10 monclient: _renew_subs 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff3a6233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff3a6233640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a01e2940 con 0x7ff3a00b45a0 2024-03-25T19:22:37.179 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff3a6233640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff3a6233640 10 monclient: _renew_subs 2024-03-25T19:22:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff3a6233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff3a6233640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a01e2cb0 con 0x7ff3a00b45a0 2024-03-25T19:22:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.175+0000 7ff3a6233640 1 librados: init done 2024-03-25T19:22:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.176+0000 7ff3a6233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.176+0000 7ff3a6233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.176+0000 7ff3a6233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.176+0000 7ff3a6233640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a004f730 con 0x7ff3a00b45a0 2024-03-25T19:22:37.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.275+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff390003610 con 0x7ff3a00b45a0 2024-03-25T19:22:37.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.275+0000 7ff387fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:37.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.275+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff390001d80 con 0x7ff3a00b45a0 2024-03-25T19:22:37.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.275+0000 7ff387fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.275+0000 7ff387fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:37.276 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.275+0000 7ff387fff640 10 monclient: dump: 2024-03-25T19:22:37.276 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.276 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.277 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.277 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.277 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.277 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.277 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.277 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.277 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.277 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.275+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff390003d30 con 0x7ff3a00b45a0 2024-03-25T19:22:37.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.275+0000 7ff39dffb640 4 set_mon_vals no callback set 2024-03-25T19:22:37.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.276+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1724631849 0 0) 0x7ff390005470 con 0x7ff3a00b45a0 2024-03-25T19:22:37.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.276+0000 7ff387fff640 10 client.4149.objecter ms_dispatch 0x7ff3a0000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:22:37.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.276+0000 7ff387fff640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:22:37.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.276+0000 7ff387fff640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:22:37.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.276+0000 7ff387fff640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-25T19:22:37.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.276+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff390004ef0 con 0x7ff3a00b45a0 2024-03-25T19:22:37.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.276+0000 7ff387fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:37.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.276+0000 7ff387fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:37.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.349+0000 7ff3a6233640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "27d55d50-50bb-48a2-8870-d038cb804e88", "id": 5}] 2024-03-25T19:22:37.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.349+0000 7ff3a6233640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "27d55d50-50bb-48a2-8870-d038cb804e88", "id": 5}] 2024-03-25T19:22:37.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.349+0000 7ff3a6233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.350 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.349+0000 7ff3a6233640 1 -- 172.21.15.151:0/2968851836 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd new", "uuid": "27d55d50-50bb-48a2-8870-d038cb804e88", "id": 5} v 0) v1 -- 0x7ff3a013b4d0 con 0x7ff3a00b45a0 2024-03-25T19:22:37.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.380+0000 7ff387fff640 1 -- 172.21.15.151:0/2968851836 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "27d55d50-50bb-48a2-8870-d038cb804e88", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1616719011 0 1988412176) 0x7ff3900051e0 con 0x7ff3a00b45a0 2024-03-25T19:22:37.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.381+0000 7ff387fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "27d55d50-50bb-48a2-8870-d038cb804e88", "id": 5}] 2024-03-25T19:22:37.381 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.381+0000 7ff387fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:37.382 INFO:teuthology.orchestra.run.smithi151.stdout:5 2024-03-25T19:22:37.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 10 librados: watch_flush enter 2024-03-25T19:22:37.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 10 librados: watch_flush exit 2024-03-25T19:22:37.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-25T19:22:37.383 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 10 client.4149.objecter successfully canceled tick 2024-03-25T19:22:37.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 10 monclient: shutdown 2024-03-25T19:22:37.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:37.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 1 -- 172.21.15.151:0/2968851836 >> v1:172.21.15.123:6789/0 conn(0x7ff3a00b45a0 legacy=0x7ff3a0144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 1 -- 172.21.15.151:0/2968851836 shutdown_connections 2024-03-25T19:22:37.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 1 -- 172.21.15.151:0/2968851836 >> 172.21.15.151:0/2968851836 conn(0x7ff3a00b7f40 msgr2=0x7ff3a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:37.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 1 -- 172.21.15.151:0/2968851836 shutdown_connections 2024-03-25T19:22:37.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 1 -- 172.21.15.151:0/2968851836 wait complete. 2024-03-25T19:22:37.384 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.382+0000 7ff3a6233640 1 librados: shutdown 2024-03-25T19:22:37.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 9b913f26-999c-4664-b1cf-49b52b0b1cda 6 2024-03-25T19:22:37.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.494+0000 7f76e1c4e640 10 monclient: get_monmap_and_config 2024-03-25T19:22:37.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.494+0000 7f76e1c4e640 10 monclient: build_initial_monmap 2024-03-25T19:22:37.495 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.494+0000 7f76e1c4e640 10 monclient: monmap: 2024-03-25T19:22:37.496 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:37.496 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.496 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:22:37.496618+0000 2024-03-25T19:22:37.496 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:22:37.496618+0000 2024-03-25T19:22:37.496 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:37.496 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.496 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:37.496 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:37.496 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:37.496 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.497 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.495+0000 7f76e1c4e640 1 Processor -- start 2024-03-25T19:22:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.495+0000 7f76e1c4e640 1 -- start start 2024-03-25T19:22:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.495+0000 7f76e1c4e640 10 monclient: init 2024-03-25T19:22:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76e1c4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76e1c4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76e1c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f76dc0b4700 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76e1c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f76dc142a20 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76e1c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f76dc0a96b0 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76e1c4e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc0a9d60 con 0x7f76dc142a20 2024-03-25T19:22:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76e1c4e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc13ff10 con 0x7f76dc0b4700 2024-03-25T19:22:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76e1c4e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc0aa1f0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76e1c4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:37.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76e1c4e640 10 monclient(hunting): authenticate will time out at 5588.780923s 2024-03-25T19:22:37.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76db7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f76dc0b4700 0x7f76dc142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37582/0 (socket says 172.21.15.151:37582) 2024-03-25T19:22:37.499 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.496+0000 7f76db7fe640 1 -- 172.21.15.151:0/206642320 learned_addr learned my addr 172.21.15.151:0/206642320 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:37.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.498+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720776855 0 0) 0x7f76dc0a9d60 con 0x7f76dc142a20 2024-03-25T19:22:37.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.498+0000 7f76daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.498+0000 7f76daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.498+0000 7f76daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.498+0000 7f76daffd640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:22:37.500 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.498+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c8003520 con 0x7f76dc142a20 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664907244 0 0) 0x7f76dc13ff10 con 0x7f76dc0b4700 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76dc0a9d60 con 0x7f76dc0b4700 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478226890 0 0) 0x7f76dc0aa1f0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.501 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:22:37.502 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.499+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76dc13ff10 con 0x7f76dc0a96b0 2024-03-25T19:22:37.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2165308009 0 0) 0x7f76c8003520 con 0x7f76dc142a20 2024-03-25T19:22:37.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 >> v1:172.21.15.123:6790/0 conn(0x7f76dc0a96b0 legacy=0x7f76dc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 >> v1:172.21.15.151:6789/0 conn(0x7f76dc0b4700 legacy=0x7f76dc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:37.503 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 1 monclient: found mon.noname-a 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76dc0b4ae0 con 0x7f76dc142a20 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f76c4001690 con 0x7f76dc142a20 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76e1c4e640 5 monclient: authenticate success, global_id 4152 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76e1c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76daffd640 10 monclient: dump: 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.505 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.505 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.505 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.505 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.505 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.505 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.505 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.505 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.502+0000 7f76e1c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:37.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76c40017f0 con 0x7f76dc142a20 2024-03-25T19:22:37.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:37.507 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76daffd640 1 -- 172.21.15.151:0/206642320 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f76c4001c00 con 0x7f76dc142a20 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76daffd640 10 monclient: dump: 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76e1c4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76e1c4e640 4 set_mon_vals no callback set 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76e1c4e640 10 monclient: shutdown 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76e1c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.504+0000 7f76e1c4e640 1 -- 172.21.15.151:0/206642320 >> v1:172.21.15.123:6789/0 conn(0x7f76dc142a20 legacy=0x7f76dc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.505+0000 7f76e1c4e640 1 -- 172.21.15.151:0/206642320 shutdown_connections 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.505+0000 7f76e1c4e640 1 -- 172.21.15.151:0/206642320 >> 172.21.15.151:0/206642320 conn(0x7f76dc0668e0 msgr2=0x7f76dc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.505+0000 7f76e1c4e640 1 -- 172.21.15.151:0/206642320 shutdown_connections 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.505+0000 7f76e1c4e640 1 -- 172.21.15.151:0/206642320 wait complete. 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.505+0000 7f76e1c4e640 10 monclient: build_initial_monmap 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.505+0000 7f76e1c4e640 10 monclient: monmap: 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:37.509 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.505+0000 7f76e1c4e640 1 librados: starting msgr at 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.505+0000 7f76e1c4e640 1 librados: starting objecter 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 1 Processor -- start 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 1 -- start start 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 1 librados: setting wanted keys 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 1 librados: calling monclient init 2024-03-25T19:22:37.510 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 10 monclient: init 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f76dc0a96b0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f76dc0b4700 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f76dc142a20 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc0adc80 con 0x7f76dc0a96b0 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc0adf60 con 0x7f76dc142a20 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc0ae240 con 0x7f76dc0b4700 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.506+0000 7f76e1c4e640 10 monclient(hunting): authenticate will time out at 5588.791236s 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.507+0000 7f76dbfff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f76dc142a20 0x7f76dc1e2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37584/0 (socket says 172.21.15.151:37584) 2024-03-25T19:22:37.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.507+0000 7f76dbfff640 1 -- 172.21.15.151:0/2499534196 learned_addr learned my addr 172.21.15.151:0/2499534196 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:37.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.507+0000 7f76d8ff9640 10 client.?.objecter ms_handle_connect 0x7f76dc142a20 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.507+0000 7f76d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.507+0000 7f76d8ff9640 10 client.?.objecter ms_handle_connect 0x7f76dc0a96b0 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.507+0000 7f76d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.507+0000 7f76d8ff9640 10 client.?.objecter ms_handle_connect 0x7f76dc0b4700 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.507+0000 7f76d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253668892 0 0) 0x7f76dc0adc80 con 0x7f76dc0a96b0 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76bc004eb0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162335070 0 0) 0x7f76dc0adf60 con 0x7f76dc142a20 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:22:37.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76dc0adc80 con 0x7f76dc142a20 2024-03-25T19:22:37.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879021479 0 0) 0x7f76dc0ae240 con 0x7f76dc0b4700 2024-03-25T19:22:37.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:22:37.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.509+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76dc0adf60 con 0x7f76dc0b4700 2024-03-25T19:22:37.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2761516823 0 0) 0x7f76bc004eb0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76dc0ae240 con 0x7f76dc0a96b0 2024-03-25T19:22:37.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f76cc001970 con 0x7f76dc0a96b0 2024-03-25T19:22:37.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:37.515 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3654785778 0 0) 0x7f76dc0adf60 con 0x7f76dc0b4700 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76bc004eb0 con 0x7f76dc0b4700 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f76c4001b00 con 0x7f76dc0b4700 2024-03-25T19:22:37.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.514+0000 7f76d8ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.517 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.516+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3654410925 0 0) 0x7f76dc0adc80 con 0x7f76dc142a20 2024-03-25T19:22:37.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.516+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76dc0adf60 con 0x7f76dc142a20 2024-03-25T19:22:37.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.516+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f76d4001c50 con 0x7f76dc142a20 2024-03-25T19:22:37.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.516+0000 7f76d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.516+0000 7f76d8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.516+0000 7f76d8ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2444164832 0 0) 0x7f76dc0ae240 con 0x7f76dc0a96b0 2024-03-25T19:22:37.520 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 >> v1:172.21.15.123:6790/0 conn(0x7f76dc0b4700 legacy=0x7f76dc1df060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 >> v1:172.21.15.151:6789/0 conn(0x7f76dc142a20 legacy=0x7f76dc1e2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 1 monclient: found mon.a 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76dc0acd50 con 0x7f76dc0a96b0 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76e1c4e640 5 monclient: authenticate success, global_id 4155 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76e1c4e640 10 monclient: _renew_subs 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76e1c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76e1c4e640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76dc0abb70 con 0x7f76dc0a96b0 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76e1c4e640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76e1c4e640 10 monclient: _renew_subs 2024-03-25T19:22:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76e1c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76e1c4e640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76dc0ac830 con 0x7f76dc0a96b0 2024-03-25T19:22:37.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.517+0000 7f76e1c4e640 1 librados: init done 2024-03-25T19:22:37.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.518+0000 7f76e1c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:37.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.518+0000 7f76e1c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:37.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.518+0000 7f76e1c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.518+0000 7f76e1c4e640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76dc04f730 con 0x7f76dc0a96b0 2024-03-25T19:22:37.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.519+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76cc0047e0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.519+0000 7f76d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:37.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.519+0000 7f76da7fc640 4 set_mon_vals no callback set 2024-03-25T19:22:37.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.611+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f76cc004af0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.611+0000 7f76d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.612 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.611+0000 7f76d8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.611+0000 7f76d8ff9640 10 monclient: dump: 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.611+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f76cc004fe0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.611+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4064644807 0 0) 0x7f76cc0068a0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.611+0000 7f76d8ff9640 10 client.4155.objecter ms_dispatch 0x7f76dc000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:22:37.613 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.611+0000 7f76d8ff9640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:22:37.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.611+0000 7f76d8ff9640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:22:37.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.611+0000 7f76d8ff9640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T19:22:37.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.612+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f76cc0063c0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.612+0000 7f76d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:37.614 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.612+0000 7f76d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:37.651 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:37.651+0000 7f327d637040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:22:37.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.682+0000 7f76e1c4e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9b913f26-999c-4664-b1cf-49b52b0b1cda", "id": 6}] 2024-03-25T19:22:37.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.682+0000 7f76e1c4e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9b913f26-999c-4664-b1cf-49b52b0b1cda", "id": 6}] 2024-03-25T19:22:37.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.682+0000 7f76e1c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.683 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.682+0000 7f76e1c4e640 1 -- 172.21.15.151:0/2499534196 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9b913f26-999c-4664-b1cf-49b52b0b1cda", "id": 6} v 0) v1 -- 0x7f76dc0b78e0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.707+0000 7f76d8ff9640 1 -- 172.21.15.151:0/2499534196 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9b913f26-999c-4664-b1cf-49b52b0b1cda", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1116777673 0 1113811849) 0x7f76cc0066b0 con 0x7f76dc0a96b0 2024-03-25T19:22:37.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.707+0000 7f76d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9b913f26-999c-4664-b1cf-49b52b0b1cda", "id": 6}] 2024-03-25T19:22:37.708 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.707+0000 7f76d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:37.708 INFO:teuthology.orchestra.run.smithi151.stdout:6 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 10 librados: watch_flush enter 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 10 librados: watch_flush exit 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 10 client.4155.objecter successfully canceled tick 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 10 monclient: shutdown 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 1 -- 172.21.15.151:0/2499534196 >> v1:172.21.15.123:6789/0 conn(0x7f76dc0a96b0 legacy=0x7f76dc1398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 1 -- 172.21.15.151:0/2499534196 shutdown_connections 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 1 -- 172.21.15.151:0/2499534196 >> 172.21.15.151:0/2499534196 conn(0x7f76dc0668e0 msgr2=0x7f76dc0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 1 -- 172.21.15.151:0/2499534196 shutdown_connections 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.708+0000 7f76e1c4e640 1 -- 172.21.15.151:0/2499534196 wait complete. 2024-03-25T19:22:37.710 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.709+0000 7f76e1c4e640 1 librados: shutdown 2024-03-25T19:22:37.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 4e7c5e0e-b326-4567-b3a5-7950666afc0c 7 2024-03-25T19:22:37.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.817+0000 7fd5e7d3b640 10 monclient: get_monmap_and_config 2024-03-25T19:22:37.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.817+0000 7fd5e7d3b640 10 monclient: build_initial_monmap 2024-03-25T19:22:37.818 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.817+0000 7fd5e7d3b640 10 monclient: monmap: 2024-03-25T19:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:22:37.819729+0000 2024-03-25T19:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:22:37.819729+0000 2024-03-25T19:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.820 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.818+0000 7fd5e7d3b640 1 Processor -- start 2024-03-25T19:22:37.820 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.818+0000 7fd5e7d3b640 1 -- start start 2024-03-25T19:22:37.820 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.818+0000 7fd5e7d3b640 10 monclient: init 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e7d3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e7d3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e7d3b640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e00b4700 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e7d3b640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e0142a20 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e7d3b640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e00a96b0 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e7d3b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e00a9d60 con 0x7fd5e00b4700 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e7d3b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e013ff10 con 0x7fd5e0142a20 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e7d3b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e00aa1f0 con 0x7fd5e00a96b0 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e7d3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e7d3b640 10 monclient(hunting): authenticate will time out at 5589.103750s 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e52d6640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd5e0142a20 0x7fd5e013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37594/0 (socket says 172.21.15.151:37594) 2024-03-25T19:22:37.821 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.819+0000 7fd5e52d6640 1 -- 172.21.15.151:0/3616532276 learned_addr learned my addr 172.21.15.151:0/3616532276 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.821+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225454075 0 0) 0x7fd5e00aa1f0 con 0x7fd5e00a96b0 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.821+0000 7fd5e4ad5640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.821+0000 7fd5e4ad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.821+0000 7fd5e4ad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.821+0000 7fd5e4ad5640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.821+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5cc003520 con 0x7fd5e00a96b0 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418302867 0 0) 0x7fd5e013ff10 con 0x7fd5e0142a20 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.823 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:22:37.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e00aa1f0 con 0x7fd5e0142a20 2024-03-25T19:22:37.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918608053 0 0) 0x7fd5e00a9d60 con 0x7fd5e00b4700 2024-03-25T19:22:37.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:22:37.824 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.822+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e013ff10 con 0x7fd5e00b4700 2024-03-25T19:22:37.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1134588868 0 0) 0x7fd5e00aa1f0 con 0x7fd5e0142a20 2024-03-25T19:22:37.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 >> v1:172.21.15.123:6790/0 conn(0x7fd5e00a96b0 legacy=0x7fd5e013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 >> v1:172.21.15.123:6789/0 conn(0x7fd5e00b4700 legacy=0x7fd5e0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 10 monclient: _finish_hunting 0 2024-03-25T19:22:37.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 1 monclient: found mon.noname-c 2024-03-25T19:22:37.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:37.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:22:37.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e00b4ae0 con 0x7fd5e0142a20 2024-03-25T19:22:37.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 10 monclient: _finish_auth 0 2024-03-25T19:22:37.826 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 10 monclient: _check_auth_tickets 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e7d3b640 5 monclient: authenticate success, global_id 4159 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd5dc001670 con 0x7fd5e0142a20 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e4ad5640 10 monclient: dump: 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.827 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.828 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.825+0000 7fd5e7d3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:37.830 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.826+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5dc0017d0 con 0x7fd5e0142a20 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.826+0000 7fd5e4ad5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.826+0000 7fd5e4ad5640 1 -- 172.21.15.151:0/3616532276 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd5dc001be0 con 0x7fd5e0142a20 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.826+0000 7fd5e7d3b640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.826+0000 7fd5e7d3b640 4 set_mon_vals no callback set 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.826+0000 7fd5e4ad5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.826+0000 7fd5e4ad5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.826+0000 7fd5e4ad5640 10 monclient: dump: 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.831 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.827+0000 7fd5e7d3b640 10 monclient: shutdown 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.827+0000 7fd5e7d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.827+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/3616532276 >> v1:172.21.15.151:6789/0 conn(0x7fd5e0142a20 legacy=0x7fd5e013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.827+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/3616532276 shutdown_connections 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.827+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/3616532276 >> 172.21.15.151:0/3616532276 conn(0x7fd5e00668e0 msgr2=0x7fd5e0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.827+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/3616532276 shutdown_connections 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.827+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/3616532276 wait complete. 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.828+0000 7fd5e7d3b640 10 monclient: build_initial_monmap 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.828+0000 7fd5e7d3b640 10 monclient: monmap: 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2024-03-25T19:22:37.832 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.828+0000 7fd5e7d3b640 1 librados: starting msgr at 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.828+0000 7fd5e7d3b640 1 librados: starting objecter 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.828+0000 7fd5e7d3b640 1 Processor -- start 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.828+0000 7fd5e7d3b640 1 -- start start 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.828+0000 7fd5e7d3b640 1 librados: setting wanted keys 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.828+0000 7fd5e7d3b640 1 librados: calling monclient init 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.828+0000 7fd5e7d3b640 10 monclient: init 2024-03-25T19:22:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e7d3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e7d3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e7d3b640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e00a96b0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e7d3b640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e00b4700 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e7d3b640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e0142a20 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e7d3b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e0141190 con 0x7fd5e00a96b0 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e7d3b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e0128340 con 0x7fd5e0142a20 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e7d3b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e014e8e0 con 0x7fd5e00b4700 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e7d3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e7d3b640 10 monclient(hunting): authenticate will time out at 5589.113712s 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e62d8640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd5e0142a20 0x7fd5e01e2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37604/0 (socket says 172.21.15.151:37604) 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5e62d8640 1 -- 172.21.15.151:0/219225667 learned_addr learned my addr 172.21.15.151:0/219225667 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5d67fc640 10 client.?.objecter ms_handle_connect 0x7fd5e0142a20 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.829+0000 7fd5d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.830+0000 7fd5d67fc640 10 client.?.objecter ms_handle_connect 0x7fd5e00a96b0 2024-03-25T19:22:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.830+0000 7fd5d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.830+0000 7fd5d67fc640 10 client.?.objecter ms_handle_connect 0x7fd5e00b4700 2024-03-25T19:22:37.835 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.830+0000 7fd5d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:37.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399185334 0 0) 0x7fd5e0128340 con 0x7fd5e0142a20 2024-03-25T19:22:37.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): my global_id is 4162 2024-03-25T19:22:37.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c4004eb0 con 0x7fd5e0142a20 2024-03-25T19:22:37.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567020155 0 0) 0x7fd5e0141190 con 0x7fd5e00a96b0 2024-03-25T19:22:37.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.836 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:22:37.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e0128340 con 0x7fd5e00a96b0 2024-03-25T19:22:37.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221766144 0 0) 0x7fd5e014e8e0 con 0x7fd5e00b4700 2024-03-25T19:22:37.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:37.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:37.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:37.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 10 monclient(hunting): my global_id is 4163 2024-03-25T19:22:37.837 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.831+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e0141190 con 0x7fd5e00b4700 2024-03-25T19:22:37.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 811497428 0 0) 0x7fd5c4004eb0 con 0x7fd5e0142a20 2024-03-25T19:22:37.838 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5e014e8e0 con 0x7fd5e0142a20 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd5d8001cd0 con 0x7fd5e0142a20 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 10 monclient(hunting): dump: 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3166460663 0 0) 0x7fd5e0128340 con 0x7fd5e00a96b0 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5c4004eb0 con 0x7fd5e00a96b0 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd5d0001870 con 0x7fd5e00a96b0 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 10 monclient(hunting): dump: 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.840 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4198904701 0 0) 0x7fd5e0141190 con 0x7fd5e00b4700 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5e0128340 con 0x7fd5e00b4700 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd5dc001ae0 con 0x7fd5e00b4700 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.836+0000 7fd5d67fc640 10 monclient(hunting): dump: 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.841 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.842 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.842 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.842 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.843 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 583876578 0 0) 0x7fd5c4004eb0 con 0x7fd5e00a96b0 2024-03-25T19:22:37.843 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 >> v1:172.21.15.123:6790/0 conn(0x7fd5e00b4700 legacy=0x7fd5e01df060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.843 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 >> v1:172.21.15.151:6789/0 conn(0x7fd5e0142a20 legacy=0x7fd5e01e2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 1 monclient: found mon.a 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e01415f0 con 0x7fd5e00a96b0 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5e7d3b640 5 monclient: authenticate success, global_id 4161 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5e7d3b640 10 monclient: _renew_subs 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5e7d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e0136810 con 0x7fd5e00a96b0 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5e7d3b640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5e7d3b640 10 monclient: _renew_subs 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5e7d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.844 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e012d910 con 0x7fd5e00a96b0 2024-03-25T19:22:37.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5e7d3b640 1 librados: init done 2024-03-25T19:22:37.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.839+0000 7fd5e7d3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:37.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.840+0000 7fd5e7d3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:37.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.840+0000 7fd5e7d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:37.845 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.840+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e004f730 con 0x7fd5e00a96b0 2024-03-25T19:22:37.936 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:37.936+0000 7f327d637040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:22:37.937 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5d0003610 con 0x7fd5e00a96b0 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd5d0003920 con 0x7fd5e00a96b0 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 10 monclient: dump: 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 3 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:37.938 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:37.939 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:22:37.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd5d0003e50 con 0x7fd5e00a96b0 2024-03-25T19:22:37.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d7fff640 4 set_mon_vals no callback set 2024-03-25T19:22:37.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 43152623 0 0) 0x7fd5d0005730 con 0x7fd5e00a96b0 2024-03-25T19:22:37.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 10 client.4161.objecter ms_dispatch 0x7fd5e0000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:22:37.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 3 client.4161.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:22:37.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 3 client.4161.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:22:37.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.937+0000 7fd5d67fc640 20 client.4161.objecter dump_active .. 0 homeless 2024-03-25T19:22:37.939 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.938+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd5d00051e0 con 0x7fd5e00a96b0 2024-03-25T19:22:37.940 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.938+0000 7fd5d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:37.940 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:37.938+0000 7fd5d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:37.992 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:37.991+0000 7f327d637040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:22:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.013+0000 7fd5e7d3b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4e7c5e0e-b326-4567-b3a5-7950666afc0c", "id": 7}] 2024-03-25T19:22:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.014+0000 7fd5e7d3b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4e7c5e0e-b326-4567-b3a5-7950666afc0c", "id": 7}] 2024-03-25T19:22:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.014+0000 7fd5e7d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.014+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/219225667 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4e7c5e0e-b326-4567-b3a5-7950666afc0c", "id": 7} v 0) v1 -- 0x7fd5e0142020 con 0x7fd5e00a96b0 2024-03-25T19:22:38.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.050+0000 7fd5d67fc640 1 -- 172.21.15.151:0/219225667 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4e7c5e0e-b326-4567-b3a5-7950666afc0c", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 342087237 0 1371665406) 0x7fd5d0005400 con 0x7fd5e00a96b0 2024-03-25T19:22:38.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.050+0000 7fd5d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4e7c5e0e-b326-4567-b3a5-7950666afc0c", "id": 7}] 2024-03-25T19:22:38.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.050+0000 7fd5d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:38.051 INFO:teuthology.orchestra.run.smithi151.stdout:7 2024-03-25T19:22:38.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.050+0000 7fd5e7d3b640 10 librados: watch_flush enter 2024-03-25T19:22:38.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.050+0000 7fd5e7d3b640 10 librados: watch_flush exit 2024-03-25T19:22:38.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.050+0000 7fd5e7d3b640 20 client.4161.objecter shutdown clearing up homeless session... 2024-03-25T19:22:38.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.050+0000 7fd5e7d3b640 10 client.4161.objecter successfully canceled tick 2024-03-25T19:22:38.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.051+0000 7fd5e7d3b640 10 monclient: shutdown 2024-03-25T19:22:38.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.051+0000 7fd5e7d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:38.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.051+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/219225667 >> v1:172.21.15.123:6789/0 conn(0x7fd5e00a96b0 legacy=0x7fd5e0127c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:38.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.051+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/219225667 shutdown_connections 2024-03-25T19:22:38.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.051+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/219225667 >> 172.21.15.151:0/219225667 conn(0x7fd5e00668e0 msgr2=0x7fd5e00b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:38.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.051+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/219225667 shutdown_connections 2024-03-25T19:22:38.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.051+0000 7fd5e7d3b640 1 -- 172.21.15.151:0/219225667 wait complete. 2024-03-25T19:22:38.053 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-25T19:22:38.051+0000 7fd5e7d3b640 1 librados: shutdown 2024-03-25T19:22:38.062 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:22:38.063 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:22:38.063 DEBUG:teuthology.orchestra.run.smithi123:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:22:38.065 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:22:38.065 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:22:38.065 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:22:38.065 DEBUG:teuthology.orchestra.run.smithi123:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:22:38.068 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:22:38.068 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:22:38.068 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:22:38.068 DEBUG:teuthology.orchestra.run.smithi123:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:22:38.070 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:22:38.070 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:22:38.070 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:22:38.071 DEBUG:teuthology.orchestra.run.smithi123:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:22:38.073 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:22:38.073 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:22:38.073 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:22:38.073 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:22:38.075 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:22:38.075 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:22:38.075 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:22:38.075 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:22:38.077 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:22:38.077 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:22:38.078 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:22:38.078 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:22:38.080 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:22:38.080 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:22:38.080 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:22:38.080 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:22:38.082 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:22:38.082 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:38.133 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:38.132+0000 7f327d637040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:22:38.174 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:38.174+0000 7f866c810040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:22:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.205+0000 7ff1f1b82640 10 monclient: get_monmap_and_config 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.205+0000 7ff1f1b82640 10 monclient: build_initial_monmap 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.206+0000 7ff1f1b82640 10 monclient: monmap: 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 0 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:22:38.206907+0000 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:22:38.206907+0000 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:38.206 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.206+0000 7ff1f1b82640 1 Processor -- start 2024-03-25T19:22:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.206+0000 7ff1f1b82640 1 -- start start 2024-03-25T19:22:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.206+0000 7ff1f1b82640 10 monclient: init 2024-03-25T19:22:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.207+0000 7ff1f1b82640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.207+0000 7ff1f1b82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.207+0000 7ff1f1b82640 10 monclient(hunting): picked mon.noname-a con 0x7ff1ec0b4700 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.207+0000 7ff1f1b82640 10 monclient(hunting): picked mon.noname-b con 0x7ff1ec142a20 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.207+0000 7ff1f1b82640 10 monclient(hunting): picked mon.noname-c con 0x7ff1ec0a96b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.207+0000 7ff1f1b82640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec0a9d60 con 0x7ff1ec0b4700 2024-03-25T19:22:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.207+0000 7ff1f1b82640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec13ff10 con 0x7ff1ec0a96b0 2024-03-25T19:22:38.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.207+0000 7ff1f1b82640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec0aa1f0 con 0x7ff1ec142a20 2024-03-25T19:22:38.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.207+0000 7ff1f1b82640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:38.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.207+0000 7ff1f1b82640 10 monclient(hunting): authenticate will time out at 5595.567501s 2024-03-25T19:22:38.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.208+0000 7ff1eaffd640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7ff1ec142a20 0x7ff1ec13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:50812/0 (socket says 172.21.15.123:50812) 2024-03-25T19:22:38.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.208+0000 7ff1eaffd640 1 -- 172.21.15.123:0/1171129255 learned_addr learned my addr 172.21.15.123:0/1171129255 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:22:38.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.210+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521689457 0 0) 0x7ff1ec0aa1f0 con 0x7ff1ec142a20 2024-03-25T19:22:38.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.210+0000 7ff1ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:38.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.210+0000 7ff1ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:38.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.210+0000 7ff1ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:38.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.210+0000 7ff1ea7fc640 10 monclient(hunting): my global_id is 4166 2024-03-25T19:22:38.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.210+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d4003520 con 0x7ff1ec142a20 2024-03-25T19:22:38.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.212+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913248990 0 0) 0x7ff1ec13ff10 con 0x7ff1ec0a96b0 2024-03-25T19:22:38.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.212+0000 7ff1ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:38.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.212+0000 7ff1ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:38.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.212+0000 7ff1ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:38.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.212+0000 7ff1ea7fc640 10 monclient(hunting): my global_id is 4165 2024-03-25T19:22:38.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.212+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ec0aa1f0 con 0x7ff1ec0a96b0 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1015506070 0 0) 0x7ff1d4003520 con 0x7ff1ec142a20 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 >> v1:172.21.15.151:6789/0 conn(0x7ff1ec0a96b0 legacy=0x7ff1ec13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 >> v1:172.21.15.123:6789/0 conn(0x7ff1ec0b4700 legacy=0x7ff1ec142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 1 monclient: found mon.noname-b 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec0b4ae0 con 0x7ff1ec142a20 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:38.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff1e0001690 con 0x7ff1ec142a20 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1ea7fc640 10 monclient: dump: 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 1 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1f1b82640 5 monclient: authenticate success, global_id 4166 2024-03-25T19:22:38.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.214+0000 7ff1f1b82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:38.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.216+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1e00017f0 con 0x7ff1ec142a20 2024-03-25T19:22:38.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.216+0000 7ff1ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:38.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.216+0000 7ff1ea7fc640 1 -- 172.21.15.123:0/1171129255 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff1e0001c00 con 0x7ff1ec142a20 2024-03-25T19:22:38.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.216+0000 7ff1ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:38.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.216+0000 7ff1ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:38.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.216+0000 7ff1ea7fc640 10 monclient: dump: 2024-03-25T19:22:38.218 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 1 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.216+0000 7ff1f1b82640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.216+0000 7ff1f1b82640 4 set_mon_vals no callback set 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.217+0000 7ff1f1b82640 10 monclient: shutdown 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.217+0000 7ff1f1b82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.217+0000 7ff1f1b82640 1 -- 172.21.15.123:0/1171129255 >> v1:172.21.15.123:6790/0 conn(0x7ff1ec142a20 legacy=0x7ff1ec13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.218+0000 7ff1f1b82640 1 -- 172.21.15.123:0/1171129255 shutdown_connections 2024-03-25T19:22:38.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.218+0000 7ff1f1b82640 1 -- 172.21.15.123:0/1171129255 >> 172.21.15.123:0/1171129255 conn(0x7ff1ec0668e0 msgr2=0x7ff1ec066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:38.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.218+0000 7ff1f1b82640 1 -- 172.21.15.123:0/1171129255 shutdown_connections 2024-03-25T19:22:38.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.218+0000 7ff1f1b82640 1 -- 172.21.15.123:0/1171129255 wait complete. 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.219+0000 7ff1f1b82640 10 monclient: build_initial_monmap 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.219+0000 7ff1f1b82640 10 monclient: monmap: 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 0 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 0.000000 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:created 0.000000 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 1 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:38.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.219+0000 7ff1f1b82640 1 librados: starting msgr at 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.219+0000 7ff1f1b82640 1 librados: starting objecter 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.219+0000 7ff1f1b82640 1 Processor -- start 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 1 -- start start 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 1 librados: setting wanted keys 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 1 librados: calling monclient init 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 10 monclient: init 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 10 monclient(hunting): picked mon.noname-a con 0x7ff1ec0a96b0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 10 monclient(hunting): picked mon.noname-c con 0x7ff1ec0b4700 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 10 monclient(hunting): picked mon.noname-b con 0x7ff1ec142a20 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec0aefc0 con 0x7ff1ec0a96b0 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.220+0000 7ff1f1b82640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec0adde0 con 0x7ff1ec142a20 2024-03-25T19:22:38.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.221+0000 7ff1f1b82640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec0adac0 con 0x7ff1ec0b4700 2024-03-25T19:22:38.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.221+0000 7ff1f1b82640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:38.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.221+0000 7ff1f1b82640 10 monclient(hunting): authenticate will time out at 5595.580882s 2024-03-25T19:22:38.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.222+0000 7ff1eaffd640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7ff1ec0b4700 0x7ff1ec130f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:50822/0 (socket says 172.21.15.123:50822) 2024-03-25T19:22:38.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.222+0000 7ff1eaffd640 1 -- 172.21.15.123:0/3746598080 learned_addr learned my addr 172.21.15.123:0/3746598080 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:22:38.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.222+0000 7ff1d3fff640 10 client.?.objecter ms_handle_connect 0x7ff1ec142a20 2024-03-25T19:22:38.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.222+0000 7ff1d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:38.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.222+0000 7ff1d3fff640 10 client.?.objecter ms_handle_connect 0x7ff1ec0b4700 2024-03-25T19:22:38.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.222+0000 7ff1d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:38.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.225+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107902049 0 0) 0x7ff1ec0adde0 con 0x7ff1ec142a20 2024-03-25T19:22:38.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.225+0000 7ff1d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:38.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.225+0000 7ff1d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:38.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.225+0000 7ff1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:38.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.225+0000 7ff1d3fff640 10 monclient(hunting): my global_id is 4168 2024-03-25T19:22:38.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.225+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d8004eb0 con 0x7ff1ec142a20 2024-03-25T19:22:38.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.226+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751530459 0 0) 0x7ff1ec0adac0 con 0x7ff1ec0b4700 2024-03-25T19:22:38.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.226+0000 7ff1d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:38.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.226+0000 7ff1d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:38.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.226+0000 7ff1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:38.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.226+0000 7ff1d3fff640 10 monclient(hunting): my global_id is 4169 2024-03-25T19:22:38.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.226+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ec0adde0 con 0x7ff1ec0b4700 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.232+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3857303483 0 0) 0x7ff1d8004eb0 con 0x7ff1ec142a20 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.232+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1ec0adac0 con 0x7ff1ec142a20 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.232+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff1e4001bd0 con 0x7ff1ec142a20 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.232+0000 7ff1d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.232+0000 7ff1d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.232+0000 7ff1d3fff640 10 monclient(hunting): dump: 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 1 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:38.233 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:38.234 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:38.234 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:38.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.232+0000 7ff1d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:38.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.234+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 28813797 0 0) 0x7ff1ec0adde0 con 0x7ff1ec0b4700 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.234+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1d8004eb0 con 0x7ff1ec0b4700 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.234+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff1e0001c00 con 0x7ff1ec0b4700 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.234+0000 7ff1d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.234+0000 7ff1d3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.234+0000 7ff1d3fff640 10 monclient(hunting): dump: 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 1 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:38.235 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:38.236 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:38.236 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:38.236 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:38.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.238+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3676279237 0 0) 0x7ff1ec0adac0 con 0x7ff1ec142a20 2024-03-25T19:22:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.238+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 >> v1:172.21.15.123:6790/0 conn(0x7ff1ec0b4700 legacy=0x7ff1ec130f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.238+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 >> v1:172.21.15.123:6789/0 conn(0x7ff1ec0a96b0 legacy=0x7ff1ec0ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.238+0000 7ff1d3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:22:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.238+0000 7ff1d3fff640 1 monclient: found mon.b 2024-03-25T19:22:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.238+0000 7ff1d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec1357f0 con 0x7ff1ec142a20 2024-03-25T19:22:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1d3fff640 10 monclient: _finish_auth 0 2024-03-25T19:22:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1d3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:22:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:38.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1f1b82640 5 monclient: authenticate success, global_id 4168 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1f1b82640 10 monclient: _renew_subs 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1f1b82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1f1b82640 1 -- 172.21.15.123:0/3746598080 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ec135aa0 con 0x7ff1ec142a20 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1f1b82640 10 client.4168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1f1b82640 10 monclient: _renew_subs 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1f1b82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1f1b82640 1 -- 172.21.15.123:0/3746598080 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ec131820 con 0x7ff1ec142a20 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.239+0000 7ff1f1b82640 1 librados: init done 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.240+0000 7ff1d1ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.241+0000 7ff1d1ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.241+0000 7ff1d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.241+0000 7ff1d1ffb640 1 -- 172.21.15.123:0/3746598080 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ec0adc50 con 0x7ff1ec142a20 2024-03-25T19:22:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1e4002080 con 0x7ff1ec142a20 2024-03-25T19:22:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff1e4003b10 con 0x7ff1ec142a20 2024-03-25T19:22:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 10 monclient: dump: 2024-03-25T19:22:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:epoch 1 2024-03-25T19:22:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:38.412 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:22:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff1e40040c0 con 0x7ff1ec142a20 2024-03-25T19:22:38.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7ff1e4005950 con 0x7ff1ec142a20 2024-03-25T19:22:38.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 10 client.4168.objecter ms_dispatch 0x7ff1ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:38.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 3 client.4168.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:38.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 3 client.4168.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:38.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1e97fa640 4 set_mon_vals no callback set 2024-03-25T19:22:38.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.411+0000 7ff1d3fff640 20 client.4168.objecter dump_active .. 0 homeless 2024-03-25T19:22:38.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.413+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff1e4005b80 con 0x7ff1ec142a20 2024-03-25T19:22:38.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.413+0000 7ff1d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:38.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.413+0000 7ff1d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:38.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.539+0000 7ff1d1ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:38.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.539+0000 7ff1d1ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:38.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.539+0000 7ff1d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:22:38.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.539+0000 7ff1d1ffb640 1 -- 172.21.15.123:0/3746598080 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1ec04f730 con 0x7ff1ec142a20 2024-03-25T19:22:38.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.608+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 4154957195 0 0) 0x7ff1e4005330 con 0x7ff1ec142a20 2024-03-25T19:22:38.671 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:38.671+0000 7f866c810040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:22:38.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.707+0000 7ff1d3fff640 1 -- 172.21.15.123:0/3746598080 <== mon.1 v1:172.21.15.151:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7ff1e4014bf0 con 0x7ff1ec142a20 2024-03-25T19:22:38.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.708+0000 7ff1d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:38.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.708+0000 7ff1d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:38.708 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-25T19:22:38.708 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:38.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 10 librados: watch_flush enter 2024-03-25T19:22:38.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 10 librados: watch_flush exit 2024-03-25T19:22:38.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 20 client.4168.objecter shutdown clearing up homeless session... 2024-03-25T19:22:38.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 10 client.4168.objecter successfully canceled tick 2024-03-25T19:22:38.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 10 monclient: shutdown 2024-03-25T19:22:38.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:38.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 1 -- 172.21.15.123:0/3746598080 >> v1:172.21.15.151:6789/0 conn(0x7ff1ec142a20 legacy=0x7ff1ec1daa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:38.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 1 -- 172.21.15.123:0/3746598080 shutdown_connections 2024-03-25T19:22:38.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 1 -- 172.21.15.123:0/3746598080 >> 172.21.15.123:0/3746598080 conn(0x7ff1ec0668e0 msgr2=0x7ff1ec0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:38.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 1 -- 172.21.15.123:0/3746598080 shutdown_connections 2024-03-25T19:22:38.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 1 -- 172.21.15.123:0/3746598080 wait complete. 2024-03-25T19:22:38.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-25T19:22:38.709+0000 7ff1d1ffb640 1 librados: shutdown 2024-03-25T19:22:38.726 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:22:38.726 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:22:38.762 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:38.762+0000 7f866c810040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:22:38.967 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-03-25T19:22:38.967+0000 7f866c810040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:22:39.532 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:39.680 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.680+0000 7f2b284dd640 10 monclient: get_monmap_and_config 2024-03-25T19:22:39.680 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.680+0000 7f2b284dd640 10 monclient: build_initial_monmap 2024-03-25T19:22:39.680 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.680+0000 7f2b284dd640 10 monclient: monmap: 2024-03-25T19:22:39.680 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:22:39.680 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:39.680 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:22:39.681203+0000 2024-03-25T19:22:39.680 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:22:39.681203+0000 2024-03-25T19:22:39.681 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:39.681 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:39.681 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:39.681 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:39.681 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:39.681 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.681+0000 7f2b284dd640 1 Processor -- start 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.681+0000 7f2b284dd640 1 -- start start 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.681+0000 7f2b284dd640 10 monclient: init 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.682+0000 7f2b284dd640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.682+0000 7f2b284dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.682+0000 7f2b284dd640 10 monclient(hunting): picked mon.noname-b con 0x7f2b200b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.682+0000 7f2b284dd640 10 monclient(hunting): picked mon.noname-a con 0x7f2b200b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.682+0000 7f2b284dd640 10 monclient(hunting): picked mon.noname-c con 0x7f2b2014be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.682+0000 7f2b284dd640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b200af940 con 0x7f2b200b1730 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.682+0000 7f2b284dd640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b200b0a70 con 0x7f2b2014be90 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.682+0000 7f2b284dd640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b200b0d50 con 0x7f2b200b4d60 2024-03-25T19:22:39.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.682+0000 7f2b284dd640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:39.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.682+0000 7f2b284dd640 10 monclient(hunting): authenticate will time out at 5597.042278s 2024-03-25T19:22:39.700 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.700+0000 7f2b25a78640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2b200b1730 0x7f2b200b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53950/0 (socket says 172.21.15.123:53950) 2024-03-25T19:22:39.700 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.700+0000 7f2b25a78640 1 -- 172.21.15.123:0/377988627 learned_addr learned my addr 172.21.15.123:0/377988627 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:22:39.914 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.915+0000 7f2b25277640 1 -- 172.21.15.123:0/377988627 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273144851 0 0) 0x7f2b200af940 con 0x7f2b200b1730 2024-03-25T19:22:39.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.915+0000 7f2b25277640 10 monclient(hunting): _negotiate 2024-03-25T19:22:39.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.915+0000 7f2b25277640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:39.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.915+0000 7f2b25277640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:39.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.915+0000 7f2b25277640 10 monclient(hunting): my global_id is 4167 2024-03-25T19:22:39.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.915+0000 7f2b25277640 1 -- 172.21.15.123:0/377988627 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b0c003520 con 0x7f2b200b1730 2024-03-25T19:22:39.918 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:39.917+0000 7f321e311640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:22:39.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 1 -- 172.21.15.123:0/377988627 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2301614095 0 0) 0x7f2b0c003520 con 0x7f2b200b1730 2024-03-25T19:22:39.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 1 -- 172.21.15.123:0/377988627 >> v1:172.21.15.123:6790/0 conn(0x7f2b200b4d60 legacy=0x7f2b2013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:39.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 1 -- 172.21.15.123:0/377988627 >> v1:172.21.15.151:6789/0 conn(0x7f2b2014be90 legacy=0x7f2b2014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:39.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 10 monclient: _finish_hunting 0 2024-03-25T19:22:39.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 1 monclient: found mon.noname-a 2024-03-25T19:22:39.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:39.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:22:39.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 1 -- 172.21.15.123:0/377988627 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b200b1030 con 0x7f2b200b1730 2024-03-25T19:22:39.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 10 monclient: _finish_auth 0 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 10 monclient: _check_auth_tickets 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b284dd640 5 monclient: authenticate success, global_id 4167 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b284dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 1 -- 172.21.15.123:0/377988627 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2b1c001670 con 0x7f2b200b1730 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b25277640 10 monclient: dump: 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:39.927 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:39.928 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:39.928 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:39.928 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:39.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.926+0000 7f2b284dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:39.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.990+0000 7f2b25277640 1 -- 172.21.15.123:0/377988627 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b1c001850 con 0x7f2b200b1730 2024-03-25T19:22:39.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.990+0000 7f2b25277640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:39.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.990+0000 7f2b25277640 1 -- 172.21.15.123:0/377988627 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2b1c001670 con 0x7f2b200b1730 2024-03-25T19:22:39.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.990+0000 7f2b25277640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:39.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.990+0000 7f2b25277640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:39.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.990+0000 7f2b25277640 10 monclient: dump: 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.990+0000 7f2b284dd640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:39.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.990+0000 7f2b284dd640 4 set_mon_vals no callback set 2024-03-25T19:22:39.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.991+0000 7f2b284dd640 10 monclient: shutdown 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.991+0000 7f2b284dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.991+0000 7f2b284dd640 1 -- 172.21.15.123:0/377988627 >> v1:172.21.15.123:6789/0 conn(0x7f2b200b1730 legacy=0x7f2b200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.991+0000 7f2b284dd640 1 -- 172.21.15.123:0/377988627 shutdown_connections 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.991+0000 7f2b284dd640 1 -- 172.21.15.123:0/377988627 >> 172.21.15.123:0/377988627 conn(0x7f2b200617a0 msgr2=0x7f2b20061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.991+0000 7f2b284dd640 1 -- 172.21.15.123:0/377988627 shutdown_connections 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.991+0000 7f2b284dd640 1 -- 172.21.15.123:0/377988627 wait complete. 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.992+0000 7f2b284dd640 10 monclient: build_initial_monmap 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.992+0000 7f2b284dd640 10 monclient: monmap: 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:22:39.993 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.992+0000 7f2b284dd640 1 librados: starting msgr at 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.992+0000 7f2b284dd640 1 librados: starting objecter 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.993+0000 7f2b284dd640 1 Processor -- start 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.993+0000 7f2b284dd640 1 -- start start 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.993+0000 7f2b284dd640 1 librados: setting wanted keys 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.993+0000 7f2b284dd640 1 librados: calling monclient init 2024-03-25T19:22:39.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.993+0000 7f2b284dd640 10 monclient: init 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.993+0000 7f2b284dd640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b284dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b284dd640 10 monclient(hunting): picked mon.noname-a con 0x7f2b200b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b284dd640 10 monclient(hunting): picked mon.noname-c con 0x7f2b200b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b284dd640 10 monclient(hunting): picked mon.noname-b con 0x7f2b2014be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b284dd640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b20137d50 con 0x7f2b200b1730 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b284dd640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b20137ee0 con 0x7f2b2014be90 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b284dd640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b20140380 con 0x7f2b200b4d60 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b284dd640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b284dd640 10 monclient(hunting): authenticate will time out at 5597.354610s 2024-03-25T19:22:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b25a78640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f2b200b4d60 0x7f2b201dae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:50832/0 (socket says 172.21.15.123:50832) 2024-03-25T19:22:39.997 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.994+0000 7f2b25a78640 1 -- 172.21.15.123:0/1104825367 learned_addr learned my addr 172.21.15.123:0/1104825367 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:22:39.997 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.995+0000 7f2b12ffd640 10 client.?.objecter ms_handle_connect 0x7f2b200b4d60 2024-03-25T19:22:39.997 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:39.995+0000 7f2b12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:40.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.001+0000 7f2b12ffd640 10 client.?.objecter ms_handle_connect 0x7f2b200b1730 2024-03-25T19:22:40.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.001+0000 7f2b12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:40.019 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:22:40.019+0000 7f321e311640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:22:40.023 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.024+0000 7f2b12ffd640 10 client.?.objecter ms_handle_connect 0x7f2b2014be90 2024-03-25T19:22:40.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.024+0000 7f2b12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:40.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.099+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578101972 0 0) 0x7f2b20137d50 con 0x7f2b200b1730 2024-03-25T19:22:40.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.099+0000 7f2b12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:40.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.099+0000 7f2b12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:40.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.099+0000 7f2b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:40.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.099+0000 7f2b12ffd640 10 monclient(hunting): my global_id is 4170 2024-03-25T19:22:40.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.099+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b04004eb0 con 0x7f2b200b1730 2024-03-25T19:22:40.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.107+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3214152330 0 0) 0x7f2b04004eb0 con 0x7f2b200b1730 2024-03-25T19:22:40.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.107+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b20137d50 con 0x7f2b200b1730 2024-03-25T19:22:40.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.109+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2b140019c0 con 0x7f2b200b1730 2024-03-25T19:22:40.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.109+0000 7f2b12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:40.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.109+0000 7f2b12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.109+0000 7f2b12ffd640 10 monclient(hunting): dump: 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:40.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.109+0000 7f2b12ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:40.133 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 891839721 0 0) 0x7f2b20137d50 con 0x7f2b200b1730 2024-03-25T19:22:40.133 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 >> v1:172.21.15.123:6790/0 conn(0x7f2b200b4d60 legacy=0x7f2b201dae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 >> v1:172.21.15.151:6789/0 conn(0x7f2b2014be90 legacy=0x7f2b201db520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 1 monclient: found mon.a 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b20140510 con 0x7f2b200b1730 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b284dd640 5 monclient: authenticate success, global_id 4170 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b284dd640 10 monclient: _renew_subs 2024-03-25T19:22:40.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b284dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:40.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b284dd640 1 -- 172.21.15.123:0/1104825367 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b20138130 con 0x7f2b200b1730 2024-03-25T19:22:40.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b284dd640 10 client.4170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:40.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b284dd640 10 monclient: _renew_subs 2024-03-25T19:22:40.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b284dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:40.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b284dd640 1 -- 172.21.15.123:0/1104825367 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b201454d0 con 0x7f2b200b1730 2024-03-25T19:22:40.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.133+0000 7f2b284dd640 1 librados: init done 2024-03-25T19:22:40.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.135+0000 7f2b284dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:40.136 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.135+0000 7f2b284dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:40.136 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.135+0000 7f2b284dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:40.136 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.135+0000 7f2b284dd640 1 -- 172.21.15.123:0/1104825367 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b200627b0 con 0x7f2b200b1730 2024-03-25T19:22:40.136 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.136+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b14003670 con 0x7f2b200b1730 2024-03-25T19:22:40.136 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.136+0000 7f2b12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:40.136 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.136+0000 7f2b24a76640 4 set_mon_vals no callback set 2024-03-25T19:22:40.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.163+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2b14003ac0 con 0x7f2b200b1730 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.163+0000 7f2b12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.163+0000 7f2b12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.163+0000 7f2b12ffd640 10 monclient: dump: 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:40.163 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:40.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.164+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 2346912251 0 0) 0x7f2b14003ca0 con 0x7f2b200b1730 2024-03-25T19:22:40.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.165+0000 7f2b25a78640 10 monclient: get_auth_request con 0x7f2b0406db30 auth_method 0 2024-03-25T19:22:40.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.165+0000 7f2b12ffd640 10 client.4170.objecter ms_handle_connect 0x7f2b0406db30 2024-03-25T19:22:40.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.191+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f2b14053650 con 0x7f2b200b1730 2024-03-25T19:22:40.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.191+0000 7f2b12ffd640 10 client.4170.objecter ms_dispatch 0x7f2b20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:40.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.191+0000 7f2b12ffd640 3 client.4170.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:40.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.191+0000 7f2b12ffd640 3 client.4170.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:40.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.191+0000 7f2b12ffd640 20 client.4170.objecter dump_active .. 0 homeless 2024-03-25T19:22:40.586 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.586+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b1404f160 con 0x7f2b200b1730 2024-03-25T19:22:40.586 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.586+0000 7f2b12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:40.586 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.586+0000 7f2b12ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:40.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.948+0000 7f2b284dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:40.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.948+0000 7f2b284dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:40.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.948+0000 7f2b284dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:40.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:40.948+0000 7f2b284dd640 1 -- 172.21.15.123:0/1104825367 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b20138370 con 0x7f2b200b1730 2024-03-25T19:22:41.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.017+0000 7f2b12ffd640 1 -- 172.21.15.123:0/1104825367 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f2b14051090 con 0x7f2b200b1730 2024-03-25T19:22:41.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.017+0000 7f2b12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:41.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.017+0000 7f2b12ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:41.017 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:22:41.017 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 10 librados: watch_flush enter 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 10 librados: watch_flush exit 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 20 client.4170.objecter shutdown clearing up homeless session... 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 10 client.4170.objecter successfully canceled tick 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 1 -- 172.21.15.123:0/1104825367 >> v1:172.21.15.151:6800/727684239 conn(0x7f2b0406db30 legacy=0x7f2b0406fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 10 monclient: shutdown 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 1 -- 172.21.15.123:0/1104825367 >> v1:172.21.15.123:6789/0 conn(0x7f2b200b1730 legacy=0x7f2b20134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 1 -- 172.21.15.123:0/1104825367 shutdown_connections 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 1 -- 172.21.15.123:0/1104825367 >> 172.21.15.123:0/1104825367 conn(0x7f2b200617a0 msgr2=0x7f2b2013be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 1 -- 172.21.15.123:0/1104825367 shutdown_connections 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 1 -- 172.21.15.123:0/1104825367 wait complete. 2024-03-25T19:22:41.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:41.020+0000 7f2b284dd640 1 librados: shutdown 2024-03-25T19:22:41.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:44.966 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:06.817 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.027 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:06.878 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.139 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:06.983 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.161 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:07.019 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.162 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:06.999 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.199 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:07.043 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.221 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:07.079 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.222 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:07.059 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.379 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:07.224 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.440 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:07.300 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.441 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:07.287 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.500 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:07.360 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.524 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:07.369 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.573 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:07.424 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.584 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:07.429 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.606 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-03-25T19:22:45.601+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:22:45.633 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:07.484 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:45.776 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-03-25T19:22:45.770+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:22:45.790 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-03-25T19:22:45.784+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:22:45.794 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-03-25T19:22:45.788+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:22:45.987 INFO:tasks.ceph.osd.1.smithi123.stderr:2024-03-25T19:22:45.981+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:22:46.051 INFO:tasks.ceph.osd.3.smithi123.stderr:2024-03-25T19:22:46.045+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:22:46.076 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:07.927 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.128 INFO:tasks.ceph.osd.0.smithi123.stderr:2024-03-25T19:22:46.122+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:22:46.233 INFO:tasks.ceph.osd.2.smithi123.stderr:2024-03-25T19:22:46.225+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:22:46.249 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:08.093 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.265 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:08.124 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.281 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:08.118 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.438 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:08.283 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.455 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:08.307 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.528 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:08.388 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.620 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:08.463 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.635 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:08.480 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.646 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:08.504 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.649 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:08.486 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.706 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:08.557 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.771 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:08.623 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.801 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:08.646 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.892 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:08.752 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:46.933 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:08.777 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.002 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:08.847 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.009 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:08.868 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.017 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:08.854 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.044 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:08.895 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.071 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:08.922 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.113 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:08.958 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.198 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:09.042 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.204 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:09.064 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.298 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:09.156 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.317 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:09.154 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.321 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:09.172 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.366 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:09.211 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.387 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:09.232 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.430 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:09.280 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.470 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:09.330 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.470 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:09.314 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.591 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:09.449 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.599 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:09.451 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.615 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:09.452 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.654 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:09.499 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.659 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:09.504 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.717 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:09.567 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.736 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:09.596 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.739 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:09.583 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.838 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:47.863 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:09.715 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.886 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:09.744 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.890 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:09.726 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.927+0000 7f81d138d640 10 monclient: get_monmap_and_config 2024-03-25T19:22:47.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.927+0000 7f81d138d640 10 monclient: build_initial_monmap 2024-03-25T19:22:47.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.927+0000 7f81d138d640 10 monclient: monmap: 2024-03-25T19:22:47.928 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:22:47.928 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:47.928 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:22:47.928692+0000 2024-03-25T19:22:47.928 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:22:47.928692+0000 2024-03-25T19:22:47.928 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:47.928 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:47.928 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:47.928 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:47.928 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:47.928 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:47.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.928+0000 7f81d138d640 1 Processor -- start 2024-03-25T19:22:47.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.928+0000 7f81d138d640 1 -- start start 2024-03-25T19:22:47.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.928+0000 7f81d138d640 10 monclient: init 2024-03-25T19:22:47.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81d138d640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81d138d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81d138d640 10 monclient(hunting): picked mon.noname-c con 0x7f81cc0b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81d138d640 10 monclient(hunting): picked mon.noname-a con 0x7f81cc0b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81d138d640 10 monclient(hunting): picked mon.noname-b con 0x7f81cc14be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81d138d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc0af940 con 0x7f81cc0b1730 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81d138d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc0b0a70 con 0x7f81cc0b4d60 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81d138d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc0b0d50 con 0x7f81cc14be90 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81d138d640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81d138d640 10 monclient(hunting): authenticate will time out at 5605.288993s 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81caffd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f81cc0b4d60 0x7f81cc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:57538/0 (socket says 172.21.15.123:57538) 2024-03-25T19:22:47.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.929+0000 7f81caffd640 1 -- 172.21.15.123:0/145321108 learned_addr learned my addr 172.21.15.123:0/145321108 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:22:47.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112495458 0 0) 0x7f81cc0b0a70 con 0x7f81cc0b4d60 2024-03-25T19:22:47.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:22:47.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b8003520 con 0x7f81cc0b4d60 2024-03-25T19:22:47.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128731559 0 0) 0x7f81cc0af940 con 0x7f81cc0b1730 2024-03-25T19:22:47.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): my global_id is 4197 2024-03-25T19:22:47.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc0b0a70 con 0x7f81cc0b1730 2024-03-25T19:22:47.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878362961 0 0) 0x7f81cc0b0d50 con 0x7f81cc14be90 2024-03-25T19:22:47.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:22:47.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.932+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc0af940 con 0x7f81cc14be90 2024-03-25T19:22:47.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2818532359 0 0) 0x7f81b8003520 con 0x7f81cc0b4d60 2024-03-25T19:22:47.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 >> v1:172.21.15.123:6790/0 conn(0x7f81cc14be90 legacy=0x7f81cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:47.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 >> v1:172.21.15.123:6789/0 conn(0x7f81cc0b1730 legacy=0x7f81cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:47.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:47.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 1 monclient: found mon.noname-c 2024-03-25T19:22:47.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc0b1030 con 0x7f81cc0b4d60 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81d138d640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81d138d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f81bc001690 con 0x7f81cc0b4d60 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81ca7fc640 10 monclient: dump: 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:47.936 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:47.937 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:47.937 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:47.937 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:47.937 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:47.937 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:47.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.935+0000 7f81d138d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:47.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81bc0017f0 con 0x7f81cc0b4d60 2024-03-25T19:22:47.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81ca7fc640 1 -- 172.21.15.123:0/145321108 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f81bc001c00 con 0x7f81cc0b4d60 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81ca7fc640 10 monclient: dump: 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:47.939 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81d138d640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81d138d640 4 set_mon_vals no callback set 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81d138d640 10 monclient: shutdown 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81d138d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81d138d640 1 -- 172.21.15.123:0/145321108 >> v1:172.21.15.151:6789/0 conn(0x7f81cc0b4d60 legacy=0x7f81cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81d138d640 1 -- 172.21.15.123:0/145321108 shutdown_connections 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81d138d640 1 -- 172.21.15.123:0/145321108 >> 172.21.15.123:0/145321108 conn(0x7f81cc0617a0 msgr2=0x7f81cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81d138d640 1 -- 172.21.15.123:0/145321108 shutdown_connections 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.937+0000 7f81d138d640 1 -- 172.21.15.123:0/145321108 wait complete. 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.938+0000 7f81d138d640 10 monclient: build_initial_monmap 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.938+0000 7f81d138d640 10 monclient: monmap: 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:22:47.940 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:47.941 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:22:47.941 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:22:47.941 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:47.941 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:22:47.941 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:47.941 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:47.941 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:47.941 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:47.941 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:09.783 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.938+0000 7f81d138d640 1 librados: starting msgr at 2024-03-25T19:22:47.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.938+0000 7f81d138d640 1 librados: starting objecter 2024-03-25T19:22:47.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.938+0000 7f81d138d640 1 Processor -- start 2024-03-25T19:22:47.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.938+0000 7f81d138d640 1 -- start start 2024-03-25T19:22:47.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.938+0000 7f81d138d640 1 librados: setting wanted keys 2024-03-25T19:22:47.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.938+0000 7f81d138d640 1 librados: calling monclient init 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.938+0000 7f81d138d640 10 monclient: init 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81d138d640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81d138d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81d138d640 10 monclient(hunting): picked mon.noname-c con 0x7f81cc0b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81d138d640 10 monclient(hunting): picked mon.noname-a con 0x7f81cc0b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81d138d640 10 monclient(hunting): picked mon.noname-b con 0x7f81cc14be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81d138d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc142050 con 0x7f81cc0b4d60 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81d138d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc1dac50 con 0x7f81cc14be90 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81d138d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc140d00 con 0x7f81cc0b1730 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81d138d640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81d138d640 10 monclient(hunting): authenticate will time out at 5605.299422s 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81caffd640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f81cc0b1730 0x7f81cc1405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:44436/0 (socket says 172.21.15.123:44436) 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.939+0000 7f81caffd640 1 -- 172.21.15.123:0/3981547676 learned_addr learned my addr 172.21.15.123:0/3981547676 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.940+0000 7f81c37fe640 10 client.?.objecter ms_handle_connect 0x7f81cc0b1730 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.940+0000 7f81c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:47.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.940+0000 7f81c37fe640 10 client.?.objecter ms_handle_connect 0x7f81cc0b4d60 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.940+0000 7f81c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.940+0000 7f81c37fe640 10 client.?.objecter ms_handle_connect 0x7f81cc14be90 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.940+0000 7f81c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967977354 0 0) 0x7f81cc1dac50 con 0x7f81cc14be90 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81ac004eb0 con 0x7f81cc14be90 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219160603 0 0) 0x7f81cc142050 con 0x7f81cc0b4d60 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): my global_id is 4200 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc1dac50 con 0x7f81cc0b4d60 2024-03-25T19:22:47.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365980427 0 0) 0x7f81cc140d00 con 0x7f81cc0b1730 2024-03-25T19:22:47.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 10 monclient(hunting): my global_id is 4223 2024-03-25T19:22:47.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.942+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc142050 con 0x7f81cc0b1730 2024-03-25T19:22:47.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 93875169 0 0) 0x7f81ac004eb0 con 0x7f81cc14be90 2024-03-25T19:22:47.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81cc140d00 con 0x7f81cc14be90 2024-03-25T19:22:47.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f81c4001bd0 con 0x7f81cc14be90 2024-03-25T19:22:47.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:47.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:47.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 10 monclient(hunting): dump: 2024-03-25T19:22:47.947 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130274749 0 0) 0x7f81cc142050 con 0x7f81cc0b1730 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81ac004eb0 con 0x7f81cc0b1730 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f81bc001b00 con 0x7f81cc0b1730 2024-03-25T19:22:47.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 10 monclient(hunting): dump: 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:47.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459409046 0 0) 0x7f81cc1dac50 con 0x7f81cc0b4d60 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81cc142050 con 0x7f81cc0b4d60 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f81b40018f0 con 0x7f81cc0b4d60 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.946+0000 7f81c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.947+0000 7f81c37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.947+0000 7f81c37fe640 10 monclient(hunting): dump: 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:47.950 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:47.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3735406047 0 0) 0x7f81cc142050 con 0x7f81cc0b4d60 2024-03-25T19:22:47.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 >> v1:172.21.15.123:6790/0 conn(0x7f81cc0b1730 legacy=0x7f81cc1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 >> v1:172.21.15.151:6789/0 conn(0x7f81cc14be90 legacy=0x7f81cc1e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 1 monclient: found mon.a 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc1423e0 con 0x7f81cc0b4d60 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 10 monclient: _finish_auth 0 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81d138d640 5 monclient: authenticate success, global_id 4200 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81d138d640 10 monclient: _renew_subs 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81d138d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:47.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81d138d640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81cc13b470 con 0x7f81cc0b4d60 2024-03-25T19:22:47.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81d138d640 10 client.4200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:47.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81d138d640 10 monclient: _renew_subs 2024-03-25T19:22:47.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81d138d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:47.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81d138d640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc1dba40 con 0x7f81cc0b4d60 2024-03-25T19:22:47.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.949+0000 7f81d138d640 1 librados: init done 2024-03-25T19:22:47.953 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:09.794 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:47.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.950+0000 7f81d138d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:47.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.950+0000 7f81d138d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:47.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.950+0000 7f81d138d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:47.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:47.950+0000 7f81d138d640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81cc1401f0 con 0x7f81cc0b4d60 2024-03-25T19:22:48.007 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:09.851 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.007 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:09.867 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.012 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:09.862 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.139 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:09.990 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.164+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81b4001d60 con 0x7f81cc0b4d60 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.164+0000 7f81c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.164+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f81b4003790 con 0x7f81cc0b4d60 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.164+0000 7f81c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.164+0000 7f81c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.164+0000 7f81c37fe640 10 monclient: dump: 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:48.164 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:48.165 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:48.165 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:48.165 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:48.165 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:48.165 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:48.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.164+0000 7f81c97fa640 4 set_mon_vals no callback set 2024-03-25T19:22:48.179 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:10.037 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.187 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:10.024 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.202 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:10.047 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.213 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.213+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f81b4019760 con 0x7f81cc0b4d60 2024-03-25T19:22:48.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.214+0000 7f81caffd640 10 monclient: get_auth_request con 0x7f81ac06e110 auth_method 0 2024-03-25T19:22:48.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.214+0000 7f81c37fe640 10 client.4200.objecter ms_handle_connect 0x7f81ac06e110 2024-03-25T19:22:48.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.228+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f81b40534d0 con 0x7f81cc0b4d60 2024-03-25T19:22:48.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.228+0000 7f81c37fe640 10 client.4200.objecter ms_dispatch 0x7f81cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:48.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.228+0000 7f81c37fe640 3 client.4200.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:48.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.228+0000 7f81c37fe640 3 client.4200.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:48.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.228+0000 7f81c37fe640 20 client.4200.objecter dump_active .. 0 homeless 2024-03-25T19:22:48.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.228+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81b404ee10 con 0x7f81cc0b4d60 2024-03-25T19:22:48.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.228+0000 7f81c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:48.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.228+0000 7f81c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:48.240 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:10.085 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.271 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:10.131 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.272 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:10.116 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.323 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:10.173 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.403 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:10.255 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.404 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.404+0000 7f81d138d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:48.404 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.404+0000 7f81d138d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:48.404 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.404+0000 7f81d138d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:48.404 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.404+0000 7f81d138d640 1 -- 172.21.15.123:0/3981547676 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81cc140e90 con 0x7f81cc0b4d60 2024-03-25T19:22:48.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.417+0000 7f81c37fe640 1 -- 172.21.15.123:0/3981547676 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f81b4050f10 con 0x7f81cc0b4d60 2024-03-25T19:22:48.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.417+0000 7f81c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:48.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.417+0000 7f81c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:48.418 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:22:48.418 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:48.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.419+0000 7f81d138d640 10 librados: watch_flush enter 2024-03-25T19:22:48.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.419+0000 7f81d138d640 10 librados: watch_flush exit 2024-03-25T19:22:48.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.419+0000 7f81d138d640 20 client.4200.objecter shutdown clearing up homeless session... 2024-03-25T19:22:48.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.419+0000 7f81d138d640 10 client.4200.objecter successfully canceled tick 2024-03-25T19:22:48.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.420+0000 7f81d138d640 1 -- 172.21.15.123:0/3981547676 >> v1:172.21.15.151:6800/727684239 conn(0x7f81ac06e110 legacy=0x7f81ac0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:48.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.420+0000 7f81d138d640 10 monclient: shutdown 2024-03-25T19:22:48.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.420+0000 7f81d138d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:48.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.420+0000 7f81d138d640 1 -- 172.21.15.123:0/3981547676 >> v1:172.21.15.123:6789/0 conn(0x7f81cc0b4d60 legacy=0x7f81cc1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:48.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.420+0000 7f81d138d640 1 -- 172.21.15.123:0/3981547676 shutdown_connections 2024-03-25T19:22:48.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.420+0000 7f81d138d640 1 -- 172.21.15.123:0/3981547676 >> 172.21.15.123:0/3981547676 conn(0x7f81cc0617a0 msgr2=0x7f81cc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:48.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.420+0000 7f81d138d640 1 -- 172.21.15.123:0/3981547676 shutdown_connections 2024-03-25T19:22:48.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.420+0000 7f81d138d640 1 -- 172.21.15.123:0/3981547676 wait complete. 2024-03-25T19:22:48.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:48.420+0000 7f81d138d640 1 librados: shutdown 2024-03-25T19:22:48.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:48.465 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:10.310 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.478 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:10.336 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.484 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:10.320 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.530 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:10.375 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.537 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:10.381 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.541 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:10.401 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.611 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:10.461 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.729 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:10.574 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.779 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:10.637 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.780 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:10.617 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.807 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:10.667 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.828 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:10.673 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.871 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:10.722 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.898 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:10.748 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:48.924 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:10.775 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.012 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:10.856 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.066 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:10.910 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.127 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:10.972 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.197 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:11.047 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.198 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:11.043 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.253 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:11.098 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.294 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:11.154 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.348 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:11.208 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.355 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:11.192 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.403 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:11.262 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.410 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:11.246 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.458 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:11.316 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.651 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:11.496 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.707 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:11.552 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.752 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:11.603 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:49.827 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:11.677 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:54.134 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:15.985 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:54.362 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:16.206 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:54.669 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:16.506 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:54.977 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:16.837 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:55.016 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:16.861 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:55.043 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:16.901 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:55.139 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:16.984 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:55.235 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:55.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.368+0000 7fa7ae5ba640 10 monclient: get_monmap_and_config 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.369+0000 7fa7ae5ba640 10 monclient: build_initial_monmap 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.369+0000 7fa7ae5ba640 10 monclient: monmap: 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:22:55.369943+0000 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:22:55.369943+0000 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:22:55.369 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:55.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.370+0000 7fa7ae5ba640 1 Processor -- start 2024-03-25T19:22:55.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.370+0000 7fa7ae5ba640 1 -- start start 2024-03-25T19:22:55.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.370+0000 7fa7ae5ba640 10 monclient: init 2024-03-25T19:22:55.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.370+0000 7fa7ae5ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:55.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.370+0000 7fa7ae5ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:55.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.370+0000 7fa7ae5ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a80a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:55.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.370+0000 7fa7ae5ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a813f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:55.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.370+0000 7fa7ae5ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a81430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:55.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.371+0000 7fa7ae5ba640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a8142520 con 0x7fa7a80a98f0 2024-03-25T19:22:55.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.371+0000 7fa7ae5ba640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a80b28a0 con 0x7fa7a81430c0 2024-03-25T19:22:55.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.371+0000 7fa7ae5ba640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a80b2a30 con 0x7fa7a813f710 2024-03-25T19:22:55.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.371+0000 7fa7ae5ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:55.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.371+0000 7fa7ae5ba640 10 monclient(hunting): authenticate will time out at 5612.730899s 2024-03-25T19:22:55.372 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.371+0000 7fa7a7fff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa7a80a98f0 0x7fa7a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:35084/0 (socket says 172.21.15.123:35084) 2024-03-25T19:22:55.372 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.371+0000 7fa7a7fff640 1 -- 172.21.15.123:0/3999325090 learned_addr learned my addr 172.21.15.123:0/3999325090 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:22:55.372 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.371+0000 7fa7a77fe640 1 --1- 172.21.15.123:0/3999325090 >> v1:172.21.15.123:6790/0 conn(0x7fa7a813f710 0x7fa7a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:44452/0 (socket says 172.21.15.123:44452) 2024-03-25T19:22:55.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135877239 0 0) 0x7fa7a80b28a0 con 0x7fa7a81430c0 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa790003520 con 0x7fa7a81430c0 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297718297 0 0) 0x7fa7a8142520 con 0x7fa7a80a98f0 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 10 monclient(hunting): my global_id is 4203 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.375+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a80b28a0 con 0x7fa7a80a98f0 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.376+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144435086 0 0) 0x7fa7a80b2a30 con 0x7fa7a813f710 2024-03-25T19:22:55.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.376+0000 7fa7a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:55.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.376+0000 7fa7a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:55.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.376+0000 7fa7a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:55.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.376+0000 7fa7a6ffd640 10 monclient(hunting): my global_id is 4226 2024-03-25T19:22:55.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.376+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a8142520 con 0x7fa7a813f710 2024-03-25T19:22:55.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3613269583 0 0) 0x7fa7a80b28a0 con 0x7fa7a80a98f0 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 >> v1:172.21.15.123:6790/0 conn(0x7fa7a813f710 legacy=0x7fa7a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 >> v1:172.21.15.151:6789/0 conn(0x7fa7a81430c0 legacy=0x7fa7a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a80b44c0 con 0x7fa7a80a98f0 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa7980015f0 con 0x7fa7a80a98f0 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7a6ffd640 10 monclient: dump: 2024-03-25T19:22:55.380 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7ae5ba640 5 monclient: authenticate success, global_id 4203 2024-03-25T19:22:55.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.380+0000 7fa7ae5ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:55.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.382+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa798001750 con 0x7fa7a80a98f0 2024-03-25T19:22:55.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.382+0000 7fa7a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:55.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.382+0000 7fa7a6ffd640 1 -- 172.21.15.123:0/3999325090 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa798001b60 con 0x7fa7a80a98f0 2024-03-25T19:22:55.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.382+0000 7fa7a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:55.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.382+0000 7fa7a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:55.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.382+0000 7fa7a6ffd640 10 monclient: dump: 2024-03-25T19:22:55.382 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:55.382 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:55.382 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.383 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.383 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:55.383 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:55.383 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:55.383 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:55.383 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:55.383 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:55.383 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.382+0000 7fa7ae5ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:55.383 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.382+0000 7fa7ae5ba640 4 set_mon_vals no callback set 2024-03-25T19:22:55.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.383+0000 7fa7ae5ba640 10 monclient: shutdown 2024-03-25T19:22:55.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.383+0000 7fa7ae5ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:55.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.383+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/3999325090 >> v1:172.21.15.123:6789/0 conn(0x7fa7a80a98f0 legacy=0x7fa7a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.383+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/3999325090 shutdown_connections 2024-03-25T19:22:55.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.383+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/3999325090 >> 172.21.15.123:0/3999325090 conn(0x7fa7a80b7ea0 msgr2=0x7fa7a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.383+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/3999325090 shutdown_connections 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.383+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/3999325090 wait complete. 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.384+0000 7fa7ae5ba640 10 monclient: build_initial_monmap 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.384+0000 7fa7ae5ba640 10 monclient: monmap: 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.384+0000 7fa7ae5ba640 1 librados: starting msgr at 2024-03-25T19:22:55.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.384+0000 7fa7ae5ba640 1 librados: starting objecter 2024-03-25T19:22:55.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.385+0000 7fa7ae5ba640 1 Processor -- start 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.385+0000 7fa7ae5ba640 1 -- start start 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.385+0000 7fa7ae5ba640 1 librados: setting wanted keys 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.385+0000 7fa7ae5ba640 1 librados: calling monclient init 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.385+0000 7fa7ae5ba640 10 monclient: init 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.386+0000 7fa7ae5ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.386+0000 7fa7ae5ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.386+0000 7fa7ae5ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a80a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.386+0000 7fa7ae5ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a813f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.386+0000 7fa7ae5ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a81430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.386+0000 7fa7ae5ba640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a8139720 con 0x7fa7a813f710 2024-03-25T19:22:55.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.386+0000 7fa7ae5ba640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a8138af0 con 0x7fa7a81430c0 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.386+0000 7fa7ae5ba640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a8138d60 con 0x7fa7a80a98f0 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.386+0000 7fa7ae5ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.386+0000 7fa7ae5ba640 10 monclient(hunting): authenticate will time out at 5612.746486s 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.387+0000 7fa7a7fff640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fa7a80a98f0 0x7fa7a8136690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:44462/0 (socket says 172.21.15.123:44462) 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.387+0000 7fa7a7fff640 1 -- 172.21.15.123:0/4086431614 learned_addr learned my addr 172.21.15.123:0/4086431614 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.387+0000 7fa7a77fe640 1 --1- 172.21.15.123:0/4086431614 >> v1:172.21.15.123:6789/0 conn(0x7fa7a813f710 0x7fa7a81372d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:35088/0 (socket says 172.21.15.123:35088) 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.387+0000 7fa7a4ff9640 10 client.?.objecter ms_handle_connect 0x7fa7a813f710 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.387+0000 7fa7a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.387+0000 7fa7a4ff9640 10 client.?.objecter ms_handle_connect 0x7fa7a80a98f0 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.387+0000 7fa7a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.387+0000 7fa7a4ff9640 10 client.?.objecter ms_handle_connect 0x7fa7a81430c0 2024-03-25T19:22:55.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.387+0000 7fa7a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.389+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229737070 0 0) 0x7fa7a8139720 con 0x7fa7a813f710 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.389+0000 7fa7a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.389+0000 7fa7a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.389+0000 7fa7a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.389+0000 7fa7a4ff9640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.389+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa794004eb0 con 0x7fa7a813f710 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.390+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878466023 0 0) 0x7fa7a8138af0 con 0x7fa7a81430c0 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.390+0000 7fa7a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.390+0000 7fa7a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.390+0000 7fa7a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.390+0000 7fa7a4ff9640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:22:55.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.390+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a8139720 con 0x7fa7a81430c0 2024-03-25T19:22:55.392 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.391+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343246318 0 0) 0x7fa7a8138d60 con 0x7fa7a80a98f0 2024-03-25T19:22:55.392 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.391+0000 7fa7a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:55.392 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.391+0000 7fa7a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:55.392 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.391+0000 7fa7a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:55.392 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.391+0000 7fa7a4ff9640 10 monclient(hunting): my global_id is 4229 2024-03-25T19:22:55.392 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.391+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a8138af0 con 0x7fa7a80a98f0 2024-03-25T19:22:55.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.393+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3843356615 0 0) 0x7fa794004eb0 con 0x7fa7a813f710 2024-03-25T19:22:55.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.393+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7a8138d60 con 0x7fa7a813f710 2024-03-25T19:22:55.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.393+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa79c0018f0 con 0x7fa7a813f710 2024-03-25T19:22:55.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.393+0000 7fa7a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:55.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.393+0000 7fa7a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:55.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.393+0000 7fa7a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:55.393 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:55.393 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:55.394 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.394 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.394 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:55.394 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:55.394 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:55.394 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:55.394 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:55.394 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:55.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.393+0000 7fa7a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:55.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.397+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4213512258 0 0) 0x7fa7a8139720 con 0x7fa7a81430c0 2024-03-25T19:22:55.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.397+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa794005b20 con 0x7fa7a81430c0 2024-03-25T19:22:55.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.397+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa7a0001b40 con 0x7fa7a81430c0 2024-03-25T19:22:55.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.397+0000 7fa7a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.397+0000 7fa7a4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.397+0000 7fa7a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:55.398 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:55.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4014524409 0 0) 0x7fa7a8138af0 con 0x7fa7a80a98f0 2024-03-25T19:22:55.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7a8139720 con 0x7fa7a80a98f0 2024-03-25T19:22:55.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa798001a60 con 0x7fa7a80a98f0 2024-03-25T19:22:55.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:55.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:55.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:55.400 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:55.400 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:55.400 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2035474510 0 0) 0x7fa7a8138d60 con 0x7fa7a813f710 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 >> v1:172.21.15.123:6790/0 conn(0x7fa7a80a98f0 legacy=0x7fa7a8136690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 >> v1:172.21.15.151:6789/0 conn(0x7fa7a81430c0 legacy=0x7fa7a80b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 1 monclient: found mon.a 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:55.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a81379e0 con 0x7fa7a813f710 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7ae5ba640 5 monclient: authenticate success, global_id 4206 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7ae5ba640 10 monclient: _renew_subs 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7ae5ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a81da8f0 con 0x7fa7a813f710 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7ae5ba640 10 client.4206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7ae5ba640 10 monclient: _renew_subs 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7ae5ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a813c380 con 0x7fa7a813f710 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.398+0000 7fa7ae5ba640 1 librados: init done 2024-03-25T19:22:55.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.400+0000 7fa7ae5ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:55.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.400+0000 7fa7ae5ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:55.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.400+0000 7fa7ae5ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:55.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.400+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a81398b0 con 0x7fa7a813f710 2024-03-25T19:22:55.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.418+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa79c0047e0 con 0x7fa7a813f710 2024-03-25T19:22:55.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.418+0000 7fa7a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:55.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.418+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa79c0018f0 con 0x7fa7a813f710 2024-03-25T19:22:55.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.418+0000 7fa7a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:55.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.418+0000 7fa7a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:55.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.418+0000 7fa7a4ff9640 10 monclient: dump: 2024-03-25T19:22:55.420 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:22:55.420 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.418+0000 7fa7a67fc640 4 set_mon_vals no callback set 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.419+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fa79c01a800 con 0x7fa7a813f710 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.419+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fa79c084530 con 0x7fa7a813f710 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.419+0000 7fa7a4ff9640 10 client.4206.objecter ms_dispatch 0x7fa7a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.419+0000 7fa7a4ff9640 3 client.4206.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:55.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.419+0000 7fa7a4ff9640 3 client.4206.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:55.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.420+0000 7fa7a4ff9640 20 client.4206.objecter dump_active .. 0 homeless 2024-03-25T19:22:55.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.420+0000 7fa7a7fff640 10 monclient: get_auth_request con 0x7fa79406e550 auth_method 0 2024-03-25T19:22:55.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.420+0000 7fa7a4ff9640 10 client.4206.objecter ms_handle_connect 0x7fa79406e550 2024-03-25T19:22:55.610 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:17.460 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:55.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.765+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa79c04fdb0 con 0x7fa7a813f710 2024-03-25T19:22:55.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.765+0000 7fa7a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:55.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.765+0000 7fa7a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:55.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.947+0000 7fa7ae5ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:55.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.948+0000 7fa7ae5ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:55.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.948+0000 7fa7ae5ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:22:55.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.948+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/4086431614 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7a8074cc0 con 0x7fa7a813f710 2024-03-25T19:22:55.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.973+0000 7fa7a4ff9640 1 -- 172.21.15.123:0/4086431614 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fa79c051eb0 con 0x7fa7a813f710 2024-03-25T19:22:55.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.973+0000 7fa7a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:55.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.973+0000 7fa7a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:55.973 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:22:55.973 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:55.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 10 librados: watch_flush enter 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 10 librados: watch_flush exit 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 20 client.4206.objecter shutdown clearing up homeless session... 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 10 client.4206.objecter successfully canceled tick 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/4086431614 >> v1:172.21.15.151:6800/727684239 conn(0x7fa79406e550 legacy=0x7fa794070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 10 monclient: shutdown 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/4086431614 >> v1:172.21.15.123:6789/0 conn(0x7fa7a813f710 legacy=0x7fa7a81372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/4086431614 shutdown_connections 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/4086431614 >> 172.21.15.123:0/4086431614 conn(0x7fa7a80b7ea0 msgr2=0x7fa7a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/4086431614 shutdown_connections 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 1 -- 172.21.15.123:0/4086431614 wait complete. 2024-03-25T19:22:55.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:22:55.975+0000 7fa7ae5ba640 1 librados: shutdown 2024-03-25T19:22:55.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:59.485 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:21.337 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:59.491 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:21.343 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:59.589 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:21.433 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:22:59.595 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:21.439 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:00.187 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:22.024 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:00.191 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:22.028 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:00.748 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:22.593 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:00.754 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:22.600 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:00.815 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:22.674 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:00.819 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:22.678 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:01.224 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:23.069 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:01.228 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:23.073 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:01.271 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:23.131 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:01.275 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:23.135 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:01.423 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:23.274 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:01.428 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:23.279 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:02.793 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:02.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.913+0000 7f62fc772640 10 monclient: get_monmap_and_config 2024-03-25T19:23:02.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.913+0000 7f62fc772640 10 monclient: build_initial_monmap 2024-03-25T19:23:02.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.913+0000 7f62fc772640 10 monclient: monmap: 2024-03-25T19:23:02.914 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:02.914 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:02.914 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:23:02.914650+0000 2024-03-25T19:23:02.914 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:23:02.914650+0000 2024-03-25T19:23:02.914 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:02.914 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:02.914 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:02.914 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:23:02.914 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:23:02.914 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.914+0000 7f62fc772640 1 Processor -- start 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.914+0000 7f62fc772640 1 -- start start 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.914+0000 7f62fc772640 10 monclient: init 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fc772640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fc772640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fc772640 10 monclient(hunting): picked mon.noname-a con 0x7f62f40b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fc772640 10 monclient(hunting): picked mon.noname-c con 0x7f62f40b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fc772640 10 monclient(hunting): picked mon.noname-b con 0x7f62f414be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fc772640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f40af940 con 0x7f62f40b4d60 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fc772640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f40b0a70 con 0x7f62f40b1730 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fc772640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f40b0d50 con 0x7f62f414be90 2024-03-25T19:23:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fc772640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:02.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fc772640 10 monclient(hunting): authenticate will time out at 5620.275724s 2024-03-25T19:23:02.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.915+0000 7f62fa50e640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f62f40b4d60 0x7f62f413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:49470/0 (socket says 172.21.15.123:49470) 2024-03-25T19:23:02.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.916+0000 7f62fa50e640 1 -- 172.21.15.123:0/3451544009 learned_addr learned my addr 172.21.15.123:0/3451544009 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:02.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306201686 0 0) 0x7f62f40b0d50 con 0x7f62f414be90 2024-03-25T19:23:02.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 10 monclient(hunting): my global_id is 4232 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e0003520 con 0x7f62f414be90 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810181652 0 0) 0x7f62f40af940 con 0x7f62f40b4d60 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f40b0d50 con 0x7f62f40b4d60 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.919+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271703372 0 0) 0x7f62f40b0a70 con 0x7f62f40b1730 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.920+0000 7f62f950c640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.920+0000 7f62f950c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.920+0000 7f62f950c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.920+0000 7f62f950c640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:23:02.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.920+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f40af940 con 0x7f62f40b1730 2024-03-25T19:23:02.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1165853193 0 0) 0x7f62f40b0d50 con 0x7f62f40b4d60 2024-03-25T19:23:02.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 >> v1:172.21.15.123:6790/0 conn(0x7f62f414be90 legacy=0x7f62f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:02.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 >> v1:172.21.15.151:6789/0 conn(0x7f62f40b1730 legacy=0x7f62f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 10 monclient: _finish_hunting 0 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 1 monclient: found mon.noname-a 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f40b1030 con 0x7f62f40b4d60 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 10 monclient: _finish_auth 0 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 10 monclient: _check_auth_tickets 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f62f0001670 con 0x7f62f40b4d60 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.922+0000 7f62f950c640 10 monclient: dump: 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:02.923 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.923+0000 7f62fc772640 5 monclient: authenticate success, global_id 4209 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.923+0000 7f62fc772640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:02.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62f00017d0 con 0x7f62f40b4d60 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62f950c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62f950c640 1 -- 172.21.15.123:0/3451544009 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f62f0001be0 con 0x7f62f40b4d60 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62fc772640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62fc772640 4 set_mon_vals no callback set 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62f950c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62f950c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62f950c640 10 monclient: dump: 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:02.925 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:02.926 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:02.926 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:02.926 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62fc772640 10 monclient: shutdown 2024-03-25T19:23:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62fc772640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.925+0000 7f62fc772640 1 -- 172.21.15.123:0/3451544009 >> v1:172.21.15.123:6789/0 conn(0x7f62f40b4d60 legacy=0x7f62f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.926+0000 7f62fc772640 1 -- 172.21.15.123:0/3451544009 shutdown_connections 2024-03-25T19:23:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.926+0000 7f62fc772640 1 -- 172.21.15.123:0/3451544009 >> 172.21.15.123:0/3451544009 conn(0x7f62f40617a0 msgr2=0x7f62f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.926+0000 7f62fc772640 1 -- 172.21.15.123:0/3451544009 shutdown_connections 2024-03-25T19:23:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.926+0000 7f62fc772640 1 -- 172.21.15.123:0/3451544009 wait complete. 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.926+0000 7f62fc772640 10 monclient: build_initial_monmap 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.926+0000 7f62fc772640 10 monclient: monmap: 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:02.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.927+0000 7f62fc772640 1 librados: starting msgr at 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.927+0000 7f62fc772640 1 librados: starting objecter 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.927+0000 7f62fc772640 1 Processor -- start 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.927+0000 7f62fc772640 1 -- start start 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.927+0000 7f62fc772640 1 librados: setting wanted keys 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.927+0000 7f62fc772640 1 librados: calling monclient init 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.927+0000 7f62fc772640 10 monclient: init 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.928+0000 7f62fc772640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.928+0000 7f62fc772640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.928+0000 7f62fc772640 10 monclient(hunting): picked mon.noname-b con 0x7f62f40b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.928+0000 7f62fc772640 10 monclient(hunting): picked mon.noname-a con 0x7f62f40b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.928+0000 7f62fc772640 10 monclient(hunting): picked mon.noname-c con 0x7f62f414be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.928+0000 7f62fc772640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f4136720 con 0x7f62f40b4d60 2024-03-25T19:23:02.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.928+0000 7f62fc772640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f41368b0 con 0x7f62f40b1730 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.928+0000 7f62fc772640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f41383d0 con 0x7f62f414be90 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.928+0000 7f62fc772640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.928+0000 7f62fc772640 10 monclient(hunting): authenticate will time out at 5620.288207s 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.929+0000 7f62f9d0d640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f62f40b4d60 0x7f62f41d7520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:49474/0 (socket says 172.21.15.123:49474) 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.929+0000 7f62f9d0d640 1 -- 172.21.15.123:0/2031388735 learned_addr learned my addr 172.21.15.123:0/2031388735 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.929+0000 7f62eaffd640 10 client.?.objecter ms_handle_connect 0x7f62f40b4d60 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.929+0000 7f62eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.929+0000 7f62eaffd640 10 client.?.objecter ms_handle_connect 0x7f62f40b1730 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.929+0000 7f62eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.929+0000 7f62eaffd640 10 client.?.objecter ms_handle_connect 0x7f62f414be90 2024-03-25T19:23:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.929+0000 7f62eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:02.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.932+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080415707 0 0) 0x7f62f41368b0 con 0x7f62f40b1730 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.932+0000 7f62eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.932+0000 7f62eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.932+0000 7f62eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.932+0000 7f62eaffd640 10 monclient(hunting): my global_id is 4228 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.932+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62d8004eb0 con 0x7f62f40b1730 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.932+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212208367 0 0) 0x7f62f4136720 con 0x7f62f40b4d60 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.932+0000 7f62eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.933+0000 7f62eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.933+0000 7f62eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.933+0000 7f62eaffd640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.933+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f41368b0 con 0x7f62f40b4d60 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.933+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798913373 0 0) 0x7f62f41383d0 con 0x7f62f414be90 2024-03-25T19:23:02.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.933+0000 7f62eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.933+0000 7f62eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.933+0000 7f62eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.933+0000 7f62eaffd640 10 monclient(hunting): my global_id is 4235 2024-03-25T19:23:02.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.933+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f4136720 con 0x7f62f414be90 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.939+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3979732226 0 0) 0x7f62d8004eb0 con 0x7f62f40b1730 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.939+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62f41383d0 con 0x7f62f40b1730 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.939+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f62f0001ba0 con 0x7f62f40b1730 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.939+0000 7f62eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.939+0000 7f62eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.939+0000 7f62eaffd640 10 monclient(hunting): dump: 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.941 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1199674610 0 0) 0x7f62f41368b0 con 0x7f62f40b4d60 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62d8004eb0 con 0x7f62f40b4d60 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f62e40019f0 con 0x7f62f40b4d60 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 10 monclient(hunting): dump: 2024-03-25T19:23:02.942 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 746126551 0 0) 0x7f62f4136720 con 0x7f62f414be90 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62f41368b0 con 0x7f62f414be90 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f62ec001a30 con 0x7f62f414be90 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:02.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.940+0000 7f62eaffd640 10 monclient(hunting): dump: 2024-03-25T19:23:02.944 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:02.944 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:02.944 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.944 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.944 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:02.944 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:02.944 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:02.944 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:02.944 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:02.944 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 584817978 0 0) 0x7f62f41383d0 con 0x7f62f40b1730 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 >> v1:172.21.15.123:6790/0 conn(0x7f62f414be90 legacy=0x7f62f41e2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 >> v1:172.21.15.123:6789/0 conn(0x7f62f40b4d60 legacy=0x7f62f41d7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 1 monclient: found mon.b 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f4138560 con 0x7f62f40b1730 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 10 monclient: _finish_auth 0 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.942+0000 7f62fc772640 5 monclient: authenticate success, global_id 4228 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.943+0000 7f62fc772640 10 monclient: _renew_subs 2024-03-25T19:23:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.943+0000 7f62fc772640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:02.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.943+0000 7f62fc772640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f4136bb0 con 0x7f62f40b1730 2024-03-25T19:23:02.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.943+0000 7f62fc772640 10 client.4228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:02.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.943+0000 7f62fc772640 10 monclient: _renew_subs 2024-03-25T19:23:02.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.943+0000 7f62fc772640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:02.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.943+0000 7f62fc772640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f413ec40 con 0x7f62f40b1730 2024-03-25T19:23:02.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.943+0000 7f62fc772640 1 librados: init done 2024-03-25T19:23:02.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.944+0000 7f62fc772640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:02.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.944+0000 7f62fc772640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:02.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.944+0000 7f62fc772640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:02.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.944+0000 7f62fc772640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f4136380 con 0x7f62f40b1730 2024-03-25T19:23:02.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.959+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62f0004a90 con 0x7f62f40b1730 2024-03-25T19:23:02.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.959+0000 7f62eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:02.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.959+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f62f0004da0 con 0x7f62f40b1730 2024-03-25T19:23:02.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.959+0000 7f62eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:02.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.959+0000 7f62eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.959+0000 7f62eaffd640 10 monclient: dump: 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:02.960 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:02.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.959+0000 7f62f8d0b640 4 set_mon_vals no callback set 2024-03-25T19:23:02.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.960+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f62f001ad70 con 0x7f62f40b1730 2024-03-25T19:23:02.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.961+0000 7f62f9d0d640 10 monclient: get_auth_request con 0x7f62d806e0f0 auth_method 0 2024-03-25T19:23:02.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.961+0000 7f62eaffd640 10 client.4228.objecter ms_handle_connect 0x7f62d806e0f0 2024-03-25T19:23:02.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.961+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f62f00548e0 con 0x7f62f40b1730 2024-03-25T19:23:02.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.962+0000 7f62eaffd640 10 client.4228.objecter ms_dispatch 0x7f62f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:02.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.962+0000 7f62eaffd640 3 client.4228.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:02.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.962+0000 7f62eaffd640 3 client.4228.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:02.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:02.962+0000 7f62eaffd640 20 client.4228.objecter dump_active .. 0 homeless 2024-03-25T19:23:03.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.328+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62f00503f0 con 0x7f62f40b1730 2024-03-25T19:23:03.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.328+0000 7f62eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:03.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.328+0000 7f62eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:03.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.529+0000 7f62fc772640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:03.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.529+0000 7f62fc772640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:03.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.529+0000 7f62fc772640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:03.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.529+0000 7f62fc772640 1 -- 172.21.15.123:0/2031388735 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62f40627b0 con 0x7f62f40b1730 2024-03-25T19:23:03.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.553+0000 7f62eaffd640 1 -- 172.21.15.123:0/2031388735 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f62f0052320 con 0x7f62f40b1730 2024-03-25T19:23:03.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.553+0000 7f62eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:03.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.553+0000 7f62eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:03.553 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:23:03.554 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:03.555 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 10 librados: watch_flush enter 2024-03-25T19:23:03.555 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 10 librados: watch_flush exit 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 20 client.4228.objecter shutdown clearing up homeless session... 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 10 client.4228.objecter successfully canceled tick 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 1 -- 172.21.15.123:0/2031388735 >> v1:172.21.15.151:6800/727684239 conn(0x7f62d806e0f0 legacy=0x7f62d80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 10 monclient: shutdown 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 1 -- 172.21.15.123:0/2031388735 >> v1:172.21.15.151:6789/0 conn(0x7f62f40b1730 legacy=0x7f62f412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 1 -- 172.21.15.123:0/2031388735 shutdown_connections 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 1 -- 172.21.15.123:0/2031388735 >> 172.21.15.123:0/2031388735 conn(0x7f62f40617a0 msgr2=0x7f62f413c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 1 -- 172.21.15.123:0/2031388735 shutdown_connections 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 1 -- 172.21.15.123:0/2031388735 wait complete. 2024-03-25T19:23:03.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:03.555+0000 7f62fc772640 1 librados: shutdown 2024-03-25T19:23:03.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:03.700 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:00:25.552 35400== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:03.974 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:00:25.818 35401== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:04.192 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:00:26.029 35402== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:04.983 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:00:26.841 35399== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:05.038 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:00:26.883 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:05.266 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:00:27.126 35423== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:05.634 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:00:27.479 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:05.989 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:00:27.839 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:23:10.372 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:10.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.511+0000 7f77269f6640 10 monclient: get_monmap_and_config 2024-03-25T19:23:10.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.511+0000 7f77269f6640 10 monclient: build_initial_monmap 2024-03-25T19:23:10.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.511+0000 7f77269f6640 10 monclient: monmap: 2024-03-25T19:23:10.512 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:10.512 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:10.512 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:23:10.512669+0000 2024-03-25T19:23:10.512 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:23:10.512669+0000 2024-03-25T19:23:10.512 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:10.512 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:10.512 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:10.512 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:23:10.512 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:23:10.512 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:10.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.512+0000 7f77269f6640 1 Processor -- start 2024-03-25T19:23:10.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.512+0000 7f77269f6640 1 -- start start 2024-03-25T19:23:10.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.512+0000 7f77269f6640 10 monclient: init 2024-03-25T19:23:10.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.513+0000 7f77269f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:10.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.513+0000 7f77269f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:10.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.513+0000 7f77269f6640 10 monclient(hunting): picked mon.noname-c con 0x7f77200b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:10.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.513+0000 7f77269f6640 10 monclient(hunting): picked mon.noname-b con 0x7f77200b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:10.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.513+0000 7f77269f6640 10 monclient(hunting): picked mon.noname-a con 0x7f772014be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.513+0000 7f77269f6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200af940 con 0x7f772014be90 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.514+0000 7f77269f6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200b0a70 con 0x7f77200b4d60 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.514+0000 7f77269f6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200b0d50 con 0x7f77200b1730 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.514+0000 7f77269f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.514+0000 7f77269f6640 10 monclient(hunting): authenticate will time out at 5627.873944s 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.514+0000 7f771f7fe640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f77200b1730 0x7f77200b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:39556/0 (socket says 172.21.15.123:39556) 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.514+0000 7f771f7fe640 1 -- 172.21.15.123:0/3329463821 learned_addr learned my addr 172.21.15.123:0/3329463821 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:10.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849988486 0 0) 0x7f77200b0d50 con 0x7f77200b1730 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 10 monclient(hunting): my global_id is 4238 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7708003520 con 0x7f77200b1730 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975434401 0 0) 0x7f77200b0a70 con 0x7f77200b4d60 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 10 monclient(hunting): my global_id is 4231 2024-03-25T19:23:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.518+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77200b0d50 con 0x7f77200b4d60 2024-03-25T19:23:10.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 281107885 0 0) 0x7f7708003520 con 0x7f77200b1730 2024-03-25T19:23:10.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 >> v1:172.21.15.151:6789/0 conn(0x7f77200b4d60 legacy=0x7f772013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:10.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 >> v1:172.21.15.123:6789/0 conn(0x7f772014be90 legacy=0x7f772014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:10.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:10.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 1 monclient: found mon.noname-b 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77200b1030 con 0x7f77200b1730 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 10 monclient: _finish_auth 0 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7714001690 con 0x7f77200b1730 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f771effd640 10 monclient: dump: 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f77269f6640 5 monclient: authenticate success, global_id 4238 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.523+0000 7f77269f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.526+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77140018b0 con 0x7f77200b1730 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.526+0000 7f771effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.526+0000 7f771effd640 1 -- 172.21.15.123:0/3329463821 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7714001c40 con 0x7f77200b1730 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.526+0000 7f77269f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.526+0000 7f77269f6640 4 set_mon_vals no callback set 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.526+0000 7f771effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.526+0000 7f771effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.526+0000 7f771effd640 10 monclient: dump: 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:10.527 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.527+0000 7f77269f6640 10 monclient: shutdown 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.527+0000 7f77269f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.527+0000 7f77269f6640 1 -- 172.21.15.123:0/3329463821 >> v1:172.21.15.123:6790/0 conn(0x7f77200b1730 legacy=0x7f77200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.527+0000 7f77269f6640 1 -- 172.21.15.123:0/3329463821 shutdown_connections 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.527+0000 7f77269f6640 1 -- 172.21.15.123:0/3329463821 >> 172.21.15.123:0/3329463821 conn(0x7f77200617a0 msgr2=0x7f7720061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.528+0000 7f77269f6640 1 -- 172.21.15.123:0/3329463821 shutdown_connections 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.528+0000 7f77269f6640 1 -- 172.21.15.123:0/3329463821 wait complete. 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.528+0000 7f77269f6640 10 monclient: build_initial_monmap 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.528+0000 7f77269f6640 10 monclient: monmap: 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:10.529 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.529+0000 7f77269f6640 1 librados: starting msgr at 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.529+0000 7f77269f6640 1 librados: starting objecter 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.529+0000 7f77269f6640 1 Processor -- start 2024-03-25T19:23:10.530 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.529+0000 7f77269f6640 1 -- start start 2024-03-25T19:23:10.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.529+0000 7f77269f6640 1 librados: setting wanted keys 2024-03-25T19:23:10.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.529+0000 7f77269f6640 1 librados: calling monclient init 2024-03-25T19:23:10.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.529+0000 7f77269f6640 10 monclient: init 2024-03-25T19:23:10.532 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f77269f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:10.532 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f77269f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:10.532 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f77269f6640 10 monclient(hunting): picked mon.noname-a con 0x7f77200b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f77269f6640 10 monclient(hunting): picked mon.noname-c con 0x7f772014be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f77269f6640 10 monclient(hunting): picked mon.noname-b con 0x7f77201399c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f77269f6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772013ae00 con 0x7f77200b4d60 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f77269f6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77201daf80 con 0x7f77201399c0 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f77269f6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77201dbbb0 con 0x7f772014be90 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f77269f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f77269f6640 10 monclient(hunting): authenticate will time out at 5627.890461s 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f771ffff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f77200b4d60 0x7f77201db4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:33204/0 (socket says 172.21.15.123:33204) 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.530+0000 7f771ffff640 1 -- 172.21.15.123:0/2011323469 learned_addr learned my addr 172.21.15.123:0/2011323469 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.531+0000 7f771cff9640 10 client.?.objecter ms_handle_connect 0x7f77200b4d60 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.531+0000 7f771cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.531+0000 7f771cff9640 10 client.?.objecter ms_handle_connect 0x7f772014be90 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.531+0000 7f771cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.531+0000 7f771cff9640 10 client.?.objecter ms_handle_connect 0x7f77201399c0 2024-03-25T19:23:10.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.531+0000 7f771cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594575638 0 0) 0x7f772013ae00 con 0x7f77200b4d60 2024-03-25T19:23:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:23:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f770c004eb0 con 0x7f77200b4d60 2024-03-25T19:23:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854124337 0 0) 0x7f77201daf80 con 0x7f77201399c0 2024-03-25T19:23:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:10.536 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 10 monclient(hunting): my global_id is 4234 2024-03-25T19:23:10.536 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.533+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f772013ae00 con 0x7f77201399c0 2024-03-25T19:23:10.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.535+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308060105 0 0) 0x7f77201dbbb0 con 0x7f772014be90 2024-03-25T19:23:10.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.535+0000 7f771cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:10.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.535+0000 7f771cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:10.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.535+0000 7f771cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:10.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.535+0000 7f771cff9640 10 monclient(hunting): my global_id is 4241 2024-03-25T19:23:10.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.535+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77201daf80 con 0x7f772014be90 2024-03-25T19:23:10.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.538+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1182518040 0 0) 0x7f770c004eb0 con 0x7f77200b4d60 2024-03-25T19:23:10.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.538+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77201dbbb0 con 0x7f77200b4d60 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.538+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7710001a60 con 0x7f77200b4d60 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.538+0000 7f771cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.538+0000 7f771cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.538+0000 7f771cff9640 10 monclient(hunting): dump: 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:10.539 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:10.540 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:10.540 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.538+0000 7f771cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:10.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108174563 0 0) 0x7f772013ae00 con 0x7f77201399c0 2024-03-25T19:23:10.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f770c004eb0 con 0x7f77201399c0 2024-03-25T19:23:10.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7718001a80 con 0x7f77201399c0 2024-03-25T19:23:10.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:10.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 10 monclient(hunting): dump: 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 716430390 0 0) 0x7f77201daf80 con 0x7f772014be90 2024-03-25T19:23:10.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f772013ae00 con 0x7f772014be90 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1580694734 0 0) 0x7f77201dbbb0 con 0x7f77200b4d60 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 >> v1:172.21.15.123:6790/0 conn(0x7f772014be90 legacy=0x7f772013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 >> v1:172.21.15.151:6789/0 conn(0x7f77201399c0 legacy=0x7f77201e3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 1 monclient: found mon.a 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77201dbd40 con 0x7f77200b4d60 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 10 monclient: _finish_auth 0 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f771cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f77269f6640 5 monclient: authenticate success, global_id 4215 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f77269f6640 10 monclient: _renew_subs 2024-03-25T19:23:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f77269f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:23:10.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f77269f6640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772013b220 con 0x7f77200b4d60 2024-03-25T19:23:10.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f77269f6640 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:10.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f77269f6640 10 monclient: _renew_subs 2024-03-25T19:23:10.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f77269f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:23:10.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.542+0000 7f77269f6640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77201396c0 con 0x7f77200b4d60 2024-03-25T19:23:10.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.543+0000 7f77269f6640 1 librados: init done 2024-03-25T19:23:10.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.544+0000 7f77269f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:10.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.544+0000 7f77269f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:10.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.544+0000 7f77269f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:23:10.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.544+0000 7f77269f6640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77201401d0 con 0x7f77200b4d60 2024-03-25T19:23:10.548 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.544+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7710001bc0 con 0x7f77200b4d60 2024-03-25T19:23:10.548 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.544+0000 7f771cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:10.548 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.544+0000 7f771e7fc640 4 set_mon_vals no callback set 2024-03-25T19:23:10.548 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.545+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7710003a40 con 0x7f77200b4d60 2024-03-25T19:23:10.548 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.545+0000 7f771cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:10.548 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.545+0000 7f771cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.545+0000 7f771cff9640 10 monclient: dump: 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:10.549 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:10.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.805+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f7710003c20 con 0x7f77200b4d60 2024-03-25T19:23:10.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.805+0000 7f771f7fe640 10 monclient: get_auth_request con 0x7f770c06e080 auth_method 0 2024-03-25T19:23:10.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.806+0000 7f771cff9640 10 client.4215.objecter ms_handle_connect 0x7f770c06e080 2024-03-25T19:23:10.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.812+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f77100548b0 con 0x7f77200b4d60 2024-03-25T19:23:10.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.812+0000 7f771cff9640 10 client.4215.objecter ms_dispatch 0x7f7720000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:10.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.812+0000 7f771cff9640 3 client.4215.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:10.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.812+0000 7f771cff9640 3 client.4215.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:10.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.812+0000 7f771cff9640 20 client.4215.objecter dump_active .. 0 homeless 2024-03-25T19:23:10.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.812+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77100501f0 con 0x7f77200b4d60 2024-03-25T19:23:10.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.812+0000 7f771cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:10.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:10.812+0000 7f771cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:11.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.165+0000 7f77269f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:11.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.165+0000 7f77269f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:11.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.165+0000 7f77269f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:23:11.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.165+0000 7f77269f6640 1 -- 172.21.15.123:0/2011323469 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f772014bcb0 con 0x7f77200b4d60 2024-03-25T19:23:11.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.179+0000 7f771cff9640 1 -- 172.21.15.123:0/2011323469 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f77100522f0 con 0x7f77200b4d60 2024-03-25T19:23:11.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.179+0000 7f771cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:11.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.179+0000 7f771cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:11.180 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:23:11.180 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.182+0000 7f77269f6640 10 librados: watch_flush enter 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.182+0000 7f77269f6640 10 librados: watch_flush exit 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.182+0000 7f77269f6640 20 client.4215.objecter shutdown clearing up homeless session... 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.182+0000 7f77269f6640 10 client.4215.objecter successfully canceled tick 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.182+0000 7f77269f6640 1 -- 172.21.15.123:0/2011323469 >> v1:172.21.15.151:6800/727684239 conn(0x7f770c06e080 legacy=0x7f770c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.182+0000 7f77269f6640 10 monclient: shutdown 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.182+0000 7f77269f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.182+0000 7f77269f6640 1 -- 172.21.15.123:0/2011323469 >> v1:172.21.15.123:6789/0 conn(0x7f77200b4d60 legacy=0x7f77201db4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.182+0000 7f77269f6640 1 -- 172.21.15.123:0/2011323469 shutdown_connections 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.182+0000 7f77269f6640 1 -- 172.21.15.123:0/2011323469 >> 172.21.15.123:0/2011323469 conn(0x7f77200617a0 msgr2=0x7f7720137580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:11.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.183+0000 7f77269f6640 1 -- 172.21.15.123:0/2011323469 shutdown_connections 2024-03-25T19:23:11.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.183+0000 7f77269f6640 1 -- 172.21.15.123:0/2011323469 wait complete. 2024-03-25T19:23:11.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:11.183+0000 7f77269f6640 1 librados: shutdown 2024-03-25T19:23:11.196 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:12.511 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-03-25T19:23:12.508+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:23:12.848 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-03-25T19:23:12.845+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:23:13.288 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-03-25T19:23:13.281+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:23:13.419 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-03-25T19:23:13.417+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:23:13.680 INFO:tasks.ceph.osd.1.smithi123.stderr:2024-03-25T19:23:13.677+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:23:14.380 INFO:tasks.ceph.osd.3.smithi123.stderr:2024-03-25T19:23:14.376+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:23:14.549 INFO:tasks.ceph.osd.0.smithi123.stderr:2024-03-25T19:23:14.547+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:23:15.053 INFO:tasks.ceph.osd.2.smithi123.stderr:2024-03-25T19:23:15.052+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:23:18.002 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:18.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.118+0000 7fb9d284a640 10 monclient: get_monmap_and_config 2024-03-25T19:23:18.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.118+0000 7fb9d284a640 10 monclient: build_initial_monmap 2024-03-25T19:23:18.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.118+0000 7fb9d284a640 10 monclient: monmap: 2024-03-25T19:23:18.118 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:18.118 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:18.118 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:23:18.119360+0000 2024-03-25T19:23:18.118 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:23:18.119360+0000 2024-03-25T19:23:18.118 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:18.119 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:18.119 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:18.119 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:23:18.119 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:23:18.119 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:18.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.119+0000 7fb9d284a640 1 Processor -- start 2024-03-25T19:23:18.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.119+0000 7fb9d284a640 1 -- start start 2024-03-25T19:23:18.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.119+0000 7fb9d284a640 10 monclient: init 2024-03-25T19:23:18.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.120+0000 7fb9d284a640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:18.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.120+0000 7fb9d284a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:18.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.120+0000 7fb9d284a640 10 monclient(hunting): picked mon.noname-a con 0x7fb9cc06c380 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:18.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.120+0000 7fb9d284a640 10 monclient(hunting): picked mon.noname-b con 0x7fb9cc0b12d0 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:18.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.120+0000 7fb9d284a640 10 monclient(hunting): picked mon.noname-c con 0x7fb9cc0aedf0 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:18.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.120+0000 7fb9d284a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc0b16f0 con 0x7fb9cc06c380 2024-03-25T19:23:18.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.120+0000 7fb9d284a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc0adfe0 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.120+0000 7fb9d284a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc0add00 con 0x7fb9cc0b12d0 2024-03-25T19:23:18.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.120+0000 7fb9d284a640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:18.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.120+0000 7fb9d284a640 10 monclient(hunting): authenticate will time out at 5635.480219s 2024-03-25T19:23:18.126 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.126+0000 7fb9cb7fe640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fb9cc0b12d0 0x7fb9cc0ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:53518/0 (socket says 172.21.15.123:53518) 2024-03-25T19:23:18.126 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.126+0000 7fb9cb7fe640 1 -- 172.21.15.123:0/378245757 learned_addr learned my addr 172.21.15.123:0/378245757 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:18.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.142+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808240769 0 0) 0x7fb9cc0add00 con 0x7fb9cc0b12d0 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.142+0000 7fb9caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.142+0000 7fb9caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.142+0000 7fb9caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.142+0000 7fb9caffd640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.142+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b4003520 con 0x7fb9cc0b12d0 2024-03-25T19:23:18.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.146+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745265043 0 0) 0x7fb9cc0b16f0 con 0x7fb9cc06c380 2024-03-25T19:23:18.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.146+0000 7fb9caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.146+0000 7fb9caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.146+0000 7fb9caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.146+0000 7fb9caffd640 10 monclient(hunting): my global_id is 4242 2024-03-25T19:23:18.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.146+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc0add00 con 0x7fb9cc06c380 2024-03-25T19:23:18.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.148+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206283275 0 0) 0x7fb9cc0adfe0 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.148+0000 7fb9caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.148+0000 7fb9caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.148+0000 7fb9caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.148+0000 7fb9caffd640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:23:18.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.148+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc0b16f0 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1745887176 0 0) 0x7fb9cc0add00 con 0x7fb9cc06c380 2024-03-25T19:23:18.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 >> v1:172.21.15.123:6790/0 conn(0x7fb9cc0b12d0 legacy=0x7fb9cc0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 >> v1:172.21.15.151:6789/0 conn(0x7fb9cc0aedf0 legacy=0x7fb9cc14e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:18.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 1 monclient: found mon.noname-a 2024-03-25T19:23:18.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc0ae2c0 con 0x7fb9cc06c380 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 10 monclient: _finish_auth 0 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb9c40019f0 con 0x7fb9cc06c380 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9caffd640 10 monclient: dump: 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.158 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:18.159 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:18.159 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:18.159 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:18.159 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:18.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9d284a640 5 monclient: authenticate success, global_id 4242 2024-03-25T19:23:18.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.157+0000 7fb9d284a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:18.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.168+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9c4001b50 con 0x7fb9cc06c380 2024-03-25T19:23:18.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.168+0000 7fb9caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.168+0000 7fb9caffd640 1 -- 172.21.15.123:0/378245757 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb9c4001ee0 con 0x7fb9cc06c380 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.168+0000 7fb9caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.168+0000 7fb9caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.168+0000 7fb9caffd640 10 monclient: dump: 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:18.169 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:18.170 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:18.170 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:18.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.169+0000 7fb9d284a640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:18.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.169+0000 7fb9d284a640 4 set_mon_vals no callback set 2024-03-25T19:23:18.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.169+0000 7fb9d284a640 10 monclient: shutdown 2024-03-25T19:23:18.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.169+0000 7fb9d284a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:18.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.169+0000 7fb9d284a640 1 -- 172.21.15.123:0/378245757 >> v1:172.21.15.123:6789/0 conn(0x7fb9cc06c380 legacy=0x7fb9cc0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.169+0000 7fb9d284a640 1 -- 172.21.15.123:0/378245757 shutdown_connections 2024-03-25T19:23:18.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.169+0000 7fb9d284a640 1 -- 172.21.15.123:0/378245757 >> 172.21.15.123:0/378245757 conn(0x7fb9cc0b5d10 msgr2=0x7fb9cc0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:18.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.169+0000 7fb9d284a640 1 -- 172.21.15.123:0/378245757 shutdown_connections 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.170+0000 7fb9d284a640 1 -- 172.21.15.123:0/378245757 wait complete. 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.170+0000 7fb9d284a640 10 monclient: build_initial_monmap 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.170+0000 7fb9d284a640 10 monclient: monmap: 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:23:18.172 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:18.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.170+0000 7fb9d284a640 1 librados: starting msgr at 2024-03-25T19:23:18.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.170+0000 7fb9d284a640 1 librados: starting objecter 2024-03-25T19:23:18.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.171+0000 7fb9d284a640 1 Processor -- start 2024-03-25T19:23:18.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.171+0000 7fb9d284a640 1 -- start start 2024-03-25T19:23:18.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.171+0000 7fb9d284a640 1 librados: setting wanted keys 2024-03-25T19:23:18.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.171+0000 7fb9d284a640 1 librados: calling monclient init 2024-03-25T19:23:18.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.171+0000 7fb9d284a640 10 monclient: init 2024-03-25T19:23:18.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.172+0000 7fb9d284a640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:18.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.172+0000 7fb9d284a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:18.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.172+0000 7fb9d284a640 10 monclient(hunting): picked mon.noname-a con 0x7fb9cc06c380 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:18.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.172+0000 7fb9d284a640 10 monclient(hunting): picked mon.noname-b con 0x7fb9cc0aedf0 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:18.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.172+0000 7fb9d284a640 10 monclient(hunting): picked mon.noname-c con 0x7fb9cc0b12d0 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:18.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.172+0000 7fb9d284a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc06c8e0 con 0x7fb9cc06c380 2024-03-25T19:23:18.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.172+0000 7fb9d284a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc1da850 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.172+0000 7fb9d284a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc1c2a30 con 0x7fb9cc0b12d0 2024-03-25T19:23:18.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.172+0000 7fb9d284a640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:18.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.172+0000 7fb9d284a640 10 monclient(hunting): authenticate will time out at 5635.532334s 2024-03-25T19:23:18.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.179+0000 7fb9cb7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fb9cc0aedf0 0x7fb9cc1dace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:34156/0 (socket says 172.21.15.123:34156) 2024-03-25T19:23:18.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.179+0000 7fb9cb7fe640 1 -- 172.21.15.123:0/3287445340 learned_addr learned my addr 172.21.15.123:0/3287445340 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:18.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.180+0000 7fb9c8ff9640 10 client.?.objecter ms_handle_connect 0x7fb9cc0aedf0 2024-03-25T19:23:18.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.180+0000 7fb9c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:18.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.185+0000 7fb9c8ff9640 10 client.?.objecter ms_handle_connect 0x7fb9cc06c380 2024-03-25T19:23:18.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.185+0000 7fb9c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:18.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.191+0000 7fb9c8ff9640 10 client.?.objecter ms_handle_connect 0x7fb9cc0b12d0 2024-03-25T19:23:18.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.191+0000 7fb9c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:18.194 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.194+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093659001 0 0) 0x7fb9cc06c8e0 con 0x7fb9cc06c380 2024-03-25T19:23:18.194 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.194+0000 7fb9c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.194+0000 7fb9c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.194+0000 7fb9c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.194+0000 7fb9c8ff9640 10 monclient(hunting): my global_id is 4245 2024-03-25T19:23:18.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.194+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b8004eb0 con 0x7fb9cc06c380 2024-03-25T19:23:18.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.202+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456132637 0 0) 0x7fb9cc1da850 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.202+0000 7fb9c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.202+0000 7fb9c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.202+0000 7fb9c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.202+0000 7fb9c8ff9640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:23:18.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.202+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc06c8e0 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.214+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 688894157 0 0) 0x7fb9cc06c8e0 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.214+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9cc1da850 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.214+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb9c0001830 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.214+0000 7fb9c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.214+0000 7fb9c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:18.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.214+0000 7fb9c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:18.214 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:18.214 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:18.214 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.215 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.215 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.215 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:18.215 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:18.215 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:18.215 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:18.215 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:18.215 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.214+0000 7fb9c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:18.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.216+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553544181 0 0) 0x7fb9cc1c2a30 con 0x7fb9cc0b12d0 2024-03-25T19:23:18.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.216+0000 7fb9c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.216+0000 7fb9c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.216+0000 7fb9c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.216+0000 7fb9c8ff9640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:23:18.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.216+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc06c8e0 con 0x7fb9cc0b12d0 2024-03-25T19:23:18.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.218+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 851967876 0 0) 0x7fb9b8004eb0 con 0x7fb9cc06c380 2024-03-25T19:23:18.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.218+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9cc1c2a30 con 0x7fb9cc06c380 2024-03-25T19:23:18.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.218+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb9c4001e60 con 0x7fb9cc06c380 2024-03-25T19:23:18.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.218+0000 7fb9c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.218+0000 7fb9c8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.218+0000 7fb9c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:18.219 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:18.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3864433430 0 0) 0x7fb9cc06c8e0 con 0x7fb9cc0b12d0 2024-03-25T19:23:18.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9b8004eb0 con 0x7fb9cc0b12d0 2024-03-25T19:23:18.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb9bc0018f0 con 0x7fb9cc0b12d0 2024-03-25T19:23:18.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:18.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:18.231 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 106948256 0 0) 0x7fb9cc1da850 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 >> v1:172.21.15.123:6790/0 conn(0x7fb9cc0b12d0 legacy=0x7fb9cc1bdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 >> v1:172.21.15.123:6789/0 conn(0x7fb9cc06c380 legacy=0x7fb9cc1c3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 1 monclient: found mon.b 2024-03-25T19:23:18.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc1ca980 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9d284a640 5 monclient: authenticate success, global_id 4264 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9d284a640 10 monclient: _renew_subs 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9d284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9d284a640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9cc1c2cd0 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9d284a640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9d284a640 10 monclient: _renew_subs 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9d284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9d284a640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc1c51b0 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.230+0000 7fb9d284a640 1 librados: init done 2024-03-25T19:23:18.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.232+0000 7fb9d284a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:18.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.232+0000 7fb9d284a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:18.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.232+0000 7fb9d284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:18.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.232+0000 7fb9d284a640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9cc1c3a30 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.512 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.512+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9c0001c60 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.512 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.512+0000 7fb9c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:18.512 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.512+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb9c0004980 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.512 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.512+0000 7fb9c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.512 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.512+0000 7fb9c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:18.512 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.512+0000 7fb9c8ff9640 10 monclient: dump: 2024-03-25T19:23:18.512 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:18.513 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:18.513 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.513 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:18.513 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.513 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:18.513 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:18.513 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:18.513 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:18.513 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:18.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.512+0000 7fb9ca7fc640 4 set_mon_vals no callback set 2024-03-25T19:23:18.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.513+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fb9c0004b60 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.513+0000 7fb9cbfff640 10 monclient: get_auth_request con 0x7fb9b806e160 auth_method 0 2024-03-25T19:23:18.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.514+0000 7fb9c8ff9640 10 client.4264.objecter ms_handle_connect 0x7fb9b806e160 2024-03-25T19:23:18.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.518+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fb9c0054740 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.518+0000 7fb9c8ff9640 10 client.4264.objecter ms_dispatch 0x7fb9cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:18.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.518+0000 7fb9c8ff9640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:18.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.518+0000 7fb9c8ff9640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:18.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.518+0000 7fb9c8ff9640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T19:23:18.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.518+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9c0050080 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.518+0000 7fb9c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:18.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.518+0000 7fb9c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:18.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.771+0000 7fb9d284a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:18.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.771+0000 7fb9d284a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:18.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.771+0000 7fb9d284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:18.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.771+0000 7fb9d284a640 1 -- 172.21.15.123:0/3287445340 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9cc1c2e60 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.808+0000 7fb9c8ff9640 1 -- 172.21.15.123:0/3287445340 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fb9c0052180 con 0x7fb9cc0aedf0 2024-03-25T19:23:18.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.808+0000 7fb9c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:18.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.808+0000 7fb9c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:18.808 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:23:18.808 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 10 librados: watch_flush enter 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 10 librados: watch_flush exit 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 10 client.4264.objecter successfully canceled tick 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 1 -- 172.21.15.123:0/3287445340 >> v1:172.21.15.151:6800/727684239 conn(0x7fb9b806e160 legacy=0x7fb9b8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 10 monclient: shutdown 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 1 -- 172.21.15.123:0/3287445340 >> v1:172.21.15.151:6789/0 conn(0x7fb9cc0aedf0 legacy=0x7fb9cc1dace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 1 -- 172.21.15.123:0/3287445340 shutdown_connections 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 1 -- 172.21.15.123:0/3287445340 >> 172.21.15.123:0/3287445340 conn(0x7fb9cc0b5d10 msgr2=0x7fb9cc1ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 1 -- 172.21.15.123:0/3287445340 shutdown_connections 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 1 -- 172.21.15.123:0/3287445340 wait complete. 2024-03-25T19:23:18.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:18.810+0000 7fb9d284a640 1 librados: shutdown 2024-03-25T19:23:18.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:25.627 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.739+0000 7fa1f789a640 10 monclient: get_monmap_and_config 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.739+0000 7fa1f789a640 10 monclient: build_initial_monmap 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.739+0000 7fa1f789a640 10 monclient: monmap: 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:23:25.741190+0000 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:23:25.741190+0000 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:25.740 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:23:25.741 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:23:25.741 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:25.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.739+0000 7fa1f789a640 1 Processor -- start 2024-03-25T19:23:25.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 1 -- start start 2024-03-25T19:23:25.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 10 monclient: init 2024-03-25T19:23:25.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:25.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:25.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f00a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:25.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f013f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f01430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f0142520 con 0x7fa1f00a98f0 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f00b28a0 con 0x7fa1f01430c0 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f00b2a30 con 0x7fa1f013f710 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.740+0000 7fa1f789a640 10 monclient(hunting): authenticate will time out at 5643.101647s 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.754+0000 7fa1f4e35640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fa1f013f710 0x7fa1f01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:35378/0 (socket says 172.21.15.123:35378) 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.754+0000 7fa1f4e35640 1 -- 172.21.15.123:0/1652180539 learned_addr learned my addr 172.21.15.123:0/1652180539 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:25.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.769+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126678722 0 0) 0x7fa1f00b28a0 con 0x7fa1f01430c0 2024-03-25T19:23:25.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:23:25.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1dc003520 con 0x7fa1f01430c0 2024-03-25T19:23:25.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838414180 0 0) 0x7fa1f0142520 con 0x7fa1f00a98f0 2024-03-25T19:23:25.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 10 monclient(hunting): my global_id is 4248 2024-03-25T19:23:25.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.770+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f00b28a0 con 0x7fa1f00a98f0 2024-03-25T19:23:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.775+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881919494 0 0) 0x7fa1f00b2a30 con 0x7fa1f013f710 2024-03-25T19:23:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.775+0000 7fa1e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.775+0000 7fa1e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.775+0000 7fa1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.775+0000 7fa1e7fff640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:23:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.775+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f0142520 con 0x7fa1f013f710 2024-03-25T19:23:25.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.787+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 558783182 0 0) 0x7fa1f0142520 con 0x7fa1f013f710 2024-03-25T19:23:25.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.787+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 >> v1:172.21.15.151:6789/0 conn(0x7fa1f01430c0 legacy=0x7fa1f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:25.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.787+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 >> v1:172.21.15.123:6789/0 conn(0x7fa1f00a98f0 legacy=0x7fa1f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:25.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.787+0000 7fa1e7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:23:25.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 1 monclient: found mon.noname-b 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f00b44c0 con 0x7fa1f013f710 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 10 monclient: _finish_auth 0 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa1e00015b0 con 0x7fa1f013f710 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1e7fff640 10 monclient: dump: 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:25.790 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:25.791 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:25.791 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:25.791 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:25.791 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:25.791 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:25.791 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:25.791 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:25.791 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:25.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1f789a640 5 monclient: authenticate success, global_id 4274 2024-03-25T19:23:25.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.788+0000 7fa1f789a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.801+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1e0001710 con 0x7fa1f013f710 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.801+0000 7fa1e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.801+0000 7fa1e7fff640 1 -- 172.21.15.123:0/1652180539 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa1e0001b20 con 0x7fa1f013f710 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.801+0000 7fa1e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.801+0000 7fa1e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.801+0000 7fa1e7fff640 10 monclient: dump: 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:25.802 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:25.803 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:25.803 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:25.803 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:25.803 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:25.803 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:25.803 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.801+0000 7fa1f789a640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:25.803 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.801+0000 7fa1f789a640 4 set_mon_vals no callback set 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.802+0000 7fa1f789a640 10 monclient: shutdown 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.802+0000 7fa1f789a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.802+0000 7fa1f789a640 1 -- 172.21.15.123:0/1652180539 >> v1:172.21.15.123:6790/0 conn(0x7fa1f013f710 legacy=0x7fa1f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.802+0000 7fa1f789a640 1 -- 172.21.15.123:0/1652180539 shutdown_connections 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.802+0000 7fa1f789a640 1 -- 172.21.15.123:0/1652180539 >> 172.21.15.123:0/1652180539 conn(0x7fa1f00b7ea0 msgr2=0x7fa1f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.802+0000 7fa1f789a640 1 -- 172.21.15.123:0/1652180539 shutdown_connections 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.802+0000 7fa1f789a640 1 -- 172.21.15.123:0/1652180539 wait complete. 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.803+0000 7fa1f789a640 10 monclient: build_initial_monmap 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.803+0000 7fa1f789a640 10 monclient: monmap: 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:25.805 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.803+0000 7fa1f789a640 1 librados: starting msgr at 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.803+0000 7fa1f789a640 1 librados: starting objecter 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.803+0000 7fa1f789a640 1 Processor -- start 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.803+0000 7fa1f789a640 1 -- start start 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.803+0000 7fa1f789a640 1 librados: setting wanted keys 2024-03-25T19:23:25.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.803+0000 7fa1f789a640 1 librados: calling monclient init 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.803+0000 7fa1f789a640 10 monclient: init 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.804+0000 7fa1f789a640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.804+0000 7fa1f789a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.804+0000 7fa1f789a640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f00a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.804+0000 7fa1f789a640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f013f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.804+0000 7fa1f789a640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f01430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.804+0000 7fa1f789a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f0138910 con 0x7fa1f01430c0 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.804+0000 7fa1f789a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f00aba70 con 0x7fa1f013f710 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.804+0000 7fa1f789a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f01381b0 con 0x7fa1f00a98f0 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.804+0000 7fa1f789a640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:25.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.804+0000 7fa1f789a640 10 monclient(hunting): authenticate will time out at 5643.165298s 2024-03-25T19:23:25.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.813+0000 7fa1f5e37640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa1f01430c0 0x7fa1f01d7ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:50916/0 (socket says 172.21.15.123:50916) 2024-03-25T19:23:25.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.813+0000 7fa1f5e37640 1 -- 172.21.15.123:0/394707746 learned_addr learned my addr 172.21.15.123:0/394707746 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:25.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.814+0000 7fa1e5ffb640 10 client.?.objecter ms_handle_connect 0x7fa1f01430c0 2024-03-25T19:23:25.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.814+0000 7fa1e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:25.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.818+0000 7fa1e5ffb640 10 client.?.objecter ms_handle_connect 0x7fa1f013f710 2024-03-25T19:23:25.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.818+0000 7fa1e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:25.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.820+0000 7fa1e5ffb640 10 client.?.objecter ms_handle_connect 0x7fa1f00a98f0 2024-03-25T19:23:25.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.820+0000 7fa1e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:25.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.821+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610031790 0 0) 0x7fa1f00aba70 con 0x7fa1f013f710 2024-03-25T19:23:25.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.821+0000 7fa1e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.821+0000 7fa1e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.821+0000 7fa1e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.821+0000 7fa1e5ffb640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:23:25.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.821+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d4004eb0 con 0x7fa1f013f710 2024-03-25T19:23:25.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.827+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715217735 0 0) 0x7fa1f0138910 con 0x7fa1f01430c0 2024-03-25T19:23:25.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.827+0000 7fa1e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.827+0000 7fa1e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.827+0000 7fa1e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.827+0000 7fa1e5ffb640 10 monclient(hunting): my global_id is 4251 2024-03-25T19:23:25.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.827+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f00aba70 con 0x7fa1f01430c0 2024-03-25T19:23:25.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.828+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2284073302 0 0) 0x7fa1d4004eb0 con 0x7fa1f013f710 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.828+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1f0138910 con 0x7fa1f013f710 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.828+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa1e0001a20 con 0x7fa1f013f710 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.828+0000 7fa1e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.828+0000 7fa1e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.828+0000 7fa1e5ffb640 10 monclient(hunting): dump: 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:25.830 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:25.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.828+0000 7fa1e5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.834+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83620150 0 0) 0x7fa1f01381b0 con 0x7fa1f00a98f0 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.834+0000 7fa1e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.834+0000 7fa1e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.834+0000 7fa1e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.834+0000 7fa1e5ffb640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.834+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d4004eb0 con 0x7fa1f00a98f0 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2959057089 0 0) 0x7fa1f0138910 con 0x7fa1f013f710 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 >> v1:172.21.15.123:6790/0 conn(0x7fa1f00a98f0 legacy=0x7fa1f01379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 >> v1:172.21.15.123:6789/0 conn(0x7fa1f01430c0 legacy=0x7fa1f01d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 1 monclient: found mon.b 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f0138340 con 0x7fa1f013f710 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1f789a640 5 monclient: authenticate success, global_id 4270 2024-03-25T19:23:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1f789a640 10 monclient: _renew_subs 2024-03-25T19:23:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1f789a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.842+0000 7fa1f789a640 1 -- 172.21.15.123:0/394707746 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f01385f0 con 0x7fa1f013f710 2024-03-25T19:23:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.843+0000 7fa1f789a640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.843+0000 7fa1f789a640 10 monclient: _renew_subs 2024-03-25T19:23:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.843+0000 7fa1f789a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.843+0000 7fa1f789a640 1 -- 172.21.15.123:0/394707746 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f01e3130 con 0x7fa1f013f710 2024-03-25T19:23:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.843+0000 7fa1f789a640 1 librados: init done 2024-03-25T19:23:25.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.844+0000 7fa1f789a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:25.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.844+0000 7fa1f789a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:25.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.844+0000 7fa1f789a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:25.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:25.844+0000 7fa1f789a640 1 -- 172.21.15.123:0/394707746 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f00627b0 con 0x7fa1f013f710 2024-03-25T19:23:26.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.153+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1e0004a90 con 0x7fa1f013f710 2024-03-25T19:23:26.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.153+0000 7fa1e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:26.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.153+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa1e0004da0 con 0x7fa1f013f710 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.153+0000 7fa1e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.153+0000 7fa1e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.153+0000 7fa1e5ffb640 10 monclient: dump: 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:26.154 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:26.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.153+0000 7fa1e77fe640 4 set_mon_vals no callback set 2024-03-25T19:23:26.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.154+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fa1e001ad70 con 0x7fa1f013f710 2024-03-25T19:23:26.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.155+0000 7fa1f5636640 10 monclient: get_auth_request con 0x7fa1d406de00 auth_method 0 2024-03-25T19:23:26.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.155+0000 7fa1e5ffb640 10 client.4270.objecter ms_handle_connect 0x7fa1d406de00 2024-03-25T19:23:26.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.159+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fa1e00547a0 con 0x7fa1f013f710 2024-03-25T19:23:26.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.159+0000 7fa1e5ffb640 10 client.4270.objecter ms_dispatch 0x7fa1f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:26.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.159+0000 7fa1e5ffb640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:26.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.159+0000 7fa1e5ffb640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:26.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.159+0000 7fa1e5ffb640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T19:23:26.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.159+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1e00502b0 con 0x7fa1f013f710 2024-03-25T19:23:26.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.160+0000 7fa1e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:26.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.160+0000 7fa1e5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:26.411 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.411+0000 7fa1f789a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:26.411 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.411+0000 7fa1f789a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:26.411 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.411+0000 7fa1f789a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:26.411 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.411+0000 7fa1f789a640 1 -- 172.21.15.123:0/394707746 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1f0069f40 con 0x7fa1f013f710 2024-03-25T19:23:26.451 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.451+0000 7fa1e5ffb640 1 -- 172.21.15.123:0/394707746 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fa1e00521e0 con 0x7fa1f013f710 2024-03-25T19:23:26.451 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.451+0000 7fa1e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:26.452 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.451+0000 7fa1e5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:26.452 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:23:26.452 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 10 librados: watch_flush enter 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 10 librados: watch_flush exit 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 10 client.4270.objecter successfully canceled tick 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 1 -- 172.21.15.123:0/394707746 >> v1:172.21.15.151:6800/727684239 conn(0x7fa1d406de00 legacy=0x7fa1d40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 10 monclient: shutdown 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 1 -- 172.21.15.123:0/394707746 >> v1:172.21.15.151:6789/0 conn(0x7fa1f013f710 legacy=0x7fa1f01d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 1 -- 172.21.15.123:0/394707746 shutdown_connections 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 1 -- 172.21.15.123:0/394707746 >> 172.21.15.123:0/394707746 conn(0x7fa1f00b7ea0 msgr2=0x7fa1f013e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 1 -- 172.21.15.123:0/394707746 shutdown_connections 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 1 -- 172.21.15.123:0/394707746 wait complete. 2024-03-25T19:23:26.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:26.454+0000 7fa1f789a640 1 librados: shutdown 2024-03-25T19:23:26.464 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:29.656 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:01:19.228 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.711 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:01:19.284 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.746 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:01:19.318 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.815 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:01:19.387 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.847 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:01:19.381 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.916 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:01:19.449 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:29.942 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:01:19.514 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:30.052 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:01:19.624 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:30.145 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:01:19.679 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:33.270 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:33.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.379+0000 7fa4aa50e640 10 monclient: get_monmap_and_config 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.379+0000 7fa4aa50e640 10 monclient: build_initial_monmap 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.380+0000 7fa4aa50e640 10 monclient: monmap: 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:23:33.380903+0000 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:23:33.380903+0000 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:23:33.380 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.380+0000 7fa4aa50e640 1 Processor -- start 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.380+0000 7fa4aa50e640 1 -- start start 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.380+0000 7fa4aa50e640 10 monclient: init 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.381+0000 7fa4aa50e640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.381+0000 7fa4aa50e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.381+0000 7fa4aa50e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a40a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.381+0000 7fa4aa50e640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.381+0000 7fa4aa50e640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a41430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.381+0000 7fa4aa50e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4142520 con 0x7fa4a40a98f0 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.381+0000 7fa4aa50e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40b28a0 con 0x7fa4a413f710 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.381+0000 7fa4aa50e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40b2a30 con 0x7fa4a41430c0 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.381+0000 7fa4aa50e640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:33.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.381+0000 7fa4aa50e640 10 monclient(hunting): authenticate will time out at 5650.741636s 2024-03-25T19:23:33.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.389+0000 7fa4a3fff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa4a40a98f0 0x7fa4a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55136/0 (socket says 172.21.15.123:55136) 2024-03-25T19:23:33.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.389+0000 7fa4a3fff640 1 -- 172.21.15.123:0/3721426095 learned_addr learned my addr 172.21.15.123:0/3721426095 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.409+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930620360 0 0) 0x7fa4a4142520 con 0x7fa4a40a98f0 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.409+0000 7fa4a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.409+0000 7fa4a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.409+0000 7fa4a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.409+0000 7fa4a2ffd640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.409+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa48c003520 con 0x7fa4a40a98f0 2024-03-25T19:23:33.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.410+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253414921 0 0) 0x7fa4a40b28a0 con 0x7fa4a413f710 2024-03-25T19:23:33.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.410+0000 7fa4a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.410+0000 7fa4a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.410+0000 7fa4a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.410+0000 7fa4a2ffd640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:23:33.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.410+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4142520 con 0x7fa4a413f710 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 464620186 0 0) 0x7fa4a4142520 con 0x7fa4a413f710 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 >> v1:172.21.15.123:6790/0 conn(0x7fa4a41430c0 legacy=0x7fa4a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 >> v1:172.21.15.123:6789/0 conn(0x7fa4a40a98f0 legacy=0x7fa4a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 1 monclient: found mon.noname-c 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a40b44c0 con 0x7fa4a413f710 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa498001670 con 0x7fa4a413f710 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4aa50e640 5 monclient: authenticate success, global_id 4273 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4aa50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4a2ffd640 10 monclient: dump: 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.420+0000 7fa4aa50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:33.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.434+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4980017d0 con 0x7fa4a413f710 2024-03-25T19:23:33.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.434+0000 7fa4a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:33.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.434+0000 7fa4a2ffd640 1 -- 172.21.15.123:0/3721426095 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa498001ba0 con 0x7fa4a413f710 2024-03-25T19:23:33.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.434+0000 7fa4aa50e640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.434+0000 7fa4aa50e640 4 set_mon_vals no callback set 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.434+0000 7fa4a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.434+0000 7fa4a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.434+0000 7fa4a2ffd640 10 monclient: dump: 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:33.435 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.435+0000 7fa4aa50e640 10 monclient: shutdown 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.435+0000 7fa4aa50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.435+0000 7fa4aa50e640 1 -- 172.21.15.123:0/3721426095 >> v1:172.21.15.151:6789/0 conn(0x7fa4a413f710 legacy=0x7fa4a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.435+0000 7fa4aa50e640 1 -- 172.21.15.123:0/3721426095 shutdown_connections 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.435+0000 7fa4aa50e640 1 -- 172.21.15.123:0/3721426095 >> 172.21.15.123:0/3721426095 conn(0x7fa4a40b7ea0 msgr2=0x7fa4a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.435+0000 7fa4aa50e640 1 -- 172.21.15.123:0/3721426095 shutdown_connections 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.435+0000 7fa4aa50e640 1 -- 172.21.15.123:0/3721426095 wait complete. 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.436+0000 7fa4aa50e640 10 monclient: build_initial_monmap 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.436+0000 7fa4aa50e640 10 monclient: monmap: 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:33.437 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.436+0000 7fa4aa50e640 1 librados: starting msgr at 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.436+0000 7fa4aa50e640 1 librados: starting objecter 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.437+0000 7fa4aa50e640 1 Processor -- start 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.437+0000 7fa4aa50e640 1 -- start start 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.437+0000 7fa4aa50e640 1 librados: setting wanted keys 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.437+0000 7fa4aa50e640 1 librados: calling monclient init 2024-03-25T19:23:33.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.437+0000 7fa4aa50e640 10 monclient: init 2024-03-25T19:23:33.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.437+0000 7fa4aa50e640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:33.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.437+0000 7fa4aa50e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:33.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.438+0000 7fa4aa50e640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a40a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:33.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.438+0000 7fa4aa50e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:33.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.438+0000 7fa4aa50e640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a41430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:33.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.438+0000 7fa4aa50e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40abfa0 con 0x7fa4a413f710 2024-03-25T19:23:33.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.438+0000 7fa4aa50e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40b1750 con 0x7fa4a40a98f0 2024-03-25T19:23:33.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.438+0000 7fa4aa50e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40ac1f0 con 0x7fa4a41430c0 2024-03-25T19:23:33.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.438+0000 7fa4aa50e640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:33.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.438+0000 7fa4aa50e640 10 monclient(hunting): authenticate will time out at 5650.798052s 2024-03-25T19:23:33.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.446+0000 7fa4a37fe640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa4a413f710 0x7fa4a4136540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:55174/0 (socket says 172.21.15.123:55174) 2024-03-25T19:23:33.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.446+0000 7fa4a37fe640 1 -- 172.21.15.123:0/533095800 learned_addr learned my addr 172.21.15.123:0/533095800 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:33.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.454+0000 7fa4a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa4a413f710 2024-03-25T19:23:33.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.454+0000 7fa4a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:33.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.457+0000 7fa4a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa4a41430c0 2024-03-25T19:23:33.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.457+0000 7fa4a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:33.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.457+0000 7fa4a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa4a40a98f0 2024-03-25T19:23:33.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.457+0000 7fa4a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:33.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.471+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245749333 0 0) 0x7fa4a40b1750 con 0x7fa4a40a98f0 2024-03-25T19:23:33.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.471+0000 7fa4a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.471+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.471+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.471+0000 7fa4a0ff9640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:23:33.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.471+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa490004eb0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.472+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145343382 0 0) 0x7fa4a40ac1f0 con 0x7fa4a41430c0 2024-03-25T19:23:33.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.472+0000 7fa4a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.472+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.472+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.472+0000 7fa4a0ff9640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:23:33.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.472+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a40b1750 con 0x7fa4a41430c0 2024-03-25T19:23:33.473 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.474+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300998365 0 0) 0x7fa4a40abfa0 con 0x7fa4a413f710 2024-03-25T19:23:33.474 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.474+0000 7fa4a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.474 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.474+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.474 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.474+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.474 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.474+0000 7fa4a0ff9640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:23:33.474 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.474+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a40ac1f0 con 0x7fa4a413f710 2024-03-25T19:23:33.492 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.492+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 913030330 0 0) 0x7fa4a40b1750 con 0x7fa4a41430c0 2024-03-25T19:23:33.492 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.492+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4a40abfa0 con 0x7fa4a41430c0 2024-03-25T19:23:33.492 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.492+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa49c001b00 con 0x7fa4a41430c0 2024-03-25T19:23:33.492 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.492+0000 7fa4a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.492 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.492+0000 7fa4a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:33.492 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.492+0000 7fa4a0ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:33.492 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:33.493 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:33.493 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.493 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.493 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.493 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:33.493 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:33.493 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:33.493 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:33.493 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:33.493 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.492+0000 7fa4a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:33.494 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3132740973 0 0) 0x7fa4a40ac1f0 con 0x7fa4a413f710 2024-03-25T19:23:33.494 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4a40b1750 con 0x7fa4a413f710 2024-03-25T19:23:33.494 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa498001530 con 0x7fa4a413f710 2024-03-25T19:23:33.494 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.494 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:33.494 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:33.494 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3731157383 0 0) 0x7fa490004eb0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4a40ac1f0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa494001950 con 0x7fa4a40a98f0 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:33.495 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.493+0000 7fa4a0ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:33.496 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:33.496 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:33.496 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.496 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.496 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.496 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:33.496 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:33.496 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:33.496 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:33.496 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:33.508 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1122123651 0 0) 0x7fa4a40ac1f0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 >> v1:172.21.15.123:6790/0 conn(0x7fa4a41430c0 legacy=0x7fa4a41388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 >> v1:172.21.15.123:6789/0 conn(0x7fa4a413f710 legacy=0x7fa4a4136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 1 monclient: found mon.b 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a41391a0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4aa50e640 5 monclient: authenticate success, global_id 4276 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.508+0000 7fa4aa50e640 10 monclient: _renew_subs 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.509+0000 7fa4aa50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.509+0000 7fa4aa50e640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a4137e80 con 0x7fa4a40a98f0 2024-03-25T19:23:33.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.509+0000 7fa4aa50e640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:33.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.509+0000 7fa4aa50e640 10 monclient: _renew_subs 2024-03-25T19:23:33.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.509+0000 7fa4aa50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:33.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.509+0000 7fa4aa50e640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a41e34b0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.509+0000 7fa4aa50e640 1 librados: init done 2024-03-25T19:23:33.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.510+0000 7fa4aa50e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:33.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.510+0000 7fa4aa50e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:33.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.510+0000 7fa4aa50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:33.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.510+0000 7fa4aa50e640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a40627b0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.792+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4940047e0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.792+0000 7fa4a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:33.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.792+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa494004af0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.792+0000 7fa4a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.792+0000 7fa4a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:33.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.792+0000 7fa4a0ff9640 10 monclient: dump: 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:33.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.792+0000 7fa4a27fc640 4 set_mon_vals no callback set 2024-03-25T19:23:33.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.793+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fa49401aac0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.794+0000 7fa4a37fe640 10 monclient: get_auth_request con 0x7fa49006e160 auth_method 0 2024-03-25T19:23:33.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.794+0000 7fa4a0ff9640 10 client.4276.objecter ms_handle_connect 0x7fa49006e160 2024-03-25T19:23:33.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.798+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fa4940547c0 con 0x7fa4a40a98f0 2024-03-25T19:23:33.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.798+0000 7fa4a0ff9640 10 client.4276.objecter ms_dispatch 0x7fa4a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:33.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.798+0000 7fa4a0ff9640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:33.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.798+0000 7fa4a0ff9640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:33.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.798+0000 7fa4a0ff9640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T19:23:33.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.799+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa494050100 con 0x7fa4a40a98f0 2024-03-25T19:23:33.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.799+0000 7fa4a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:33.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:33.799+0000 7fa4a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:34.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.048+0000 7fa4aa50e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:34.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.048+0000 7fa4aa50e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:34.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.048+0000 7fa4aa50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:34.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.048+0000 7fa4aa50e640 1 -- 172.21.15.123:0/533095800 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4a41393e0 con 0x7fa4a40a98f0 2024-03-25T19:23:34.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.086+0000 7fa4a0ff9640 1 -- 172.21.15.123:0/533095800 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fa4a41393e0 con 0x7fa4a40a98f0 2024-03-25T19:23:34.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.087+0000 7fa4a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:34.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.087+0000 7fa4a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:34.087 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:23:34.087 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:34.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 10 librados: watch_flush enter 2024-03-25T19:23:34.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 10 librados: watch_flush exit 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 10 client.4276.objecter successfully canceled tick 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 1 -- 172.21.15.123:0/533095800 >> v1:172.21.15.151:6800/727684239 conn(0x7fa49006e160 legacy=0x7fa490070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 10 monclient: shutdown 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 1 -- 172.21.15.123:0/533095800 >> v1:172.21.15.151:6789/0 conn(0x7fa4a40a98f0 legacy=0x7fa4a40ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 1 -- 172.21.15.123:0/533095800 shutdown_connections 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 1 -- 172.21.15.123:0/533095800 >> 172.21.15.123:0/533095800 conn(0x7fa4a40b7ea0 msgr2=0x7fa4a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 1 -- 172.21.15.123:0/533095800 shutdown_connections 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 1 -- 172.21.15.123:0/533095800 wait complete. 2024-03-25T19:23:34.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:34.089+0000 7fa4aa50e640 1 librados: shutdown 2024-03-25T19:23:34.102 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:40.908 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:41.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.013+0000 7fc56fe1e640 10 monclient: get_monmap_and_config 2024-03-25T19:23:41.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.013+0000 7fc56fe1e640 10 monclient: build_initial_monmap 2024-03-25T19:23:41.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.013+0000 7fc56fe1e640 10 monclient: monmap: 2024-03-25T19:23:41.014 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:41.014 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:41.014 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:23:41.014743+0000 2024-03-25T19:23:41.014 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:23:41.014743+0000 2024-03-25T19:23:41.014 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:41.014 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:41.014 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:41.014 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:23:41.014 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:23:41.014 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.014+0000 7fc56fe1e640 1 Processor -- start 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.014+0000 7fc56fe1e640 1 -- start start 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.014+0000 7fc56fe1e640 10 monclient: init 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56fe1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56fe1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56fe1e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5680a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56fe1e640 10 monclient(hunting): picked mon.noname-c con 0x7fc56813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56fe1e640 10 monclient(hunting): picked mon.noname-b con 0x7fc5681430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56fe1e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc568142520 con 0x7fc5680a98f0 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56fe1e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5680b28a0 con 0x7fc56813f710 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56fe1e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5680b2a30 con 0x7fc5681430c0 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56fe1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56fe1e640 10 monclient(hunting): authenticate will time out at 5658.375161s 2024-03-25T19:23:41.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56e3bb640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fc5681430c0 0x7fc56814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:37902/0 (socket says 172.21.15.123:37902) 2024-03-25T19:23:41.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.015+0000 7fc56e3bb640 1 -- 172.21.15.123:0/31719327 learned_addr learned my addr 172.21.15.123:0/31719327 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.088+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243070219 0 0) 0x7fc5680b28a0 con 0x7fc56813f710 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.088+0000 7fc56cbb8640 10 monclient(hunting): _negotiate 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.088+0000 7fc56cbb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.088+0000 7fc56cbb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.088+0000 7fc56cbb8640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.088+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc550003520 con 0x7fc56813f710 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270624339 0 0) 0x7fc568142520 con 0x7fc5680a98f0 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 10 monclient(hunting): _negotiate 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5680b28a0 con 0x7fc5680a98f0 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749243709 0 0) 0x7fc5680b2a30 con 0x7fc5681430c0 2024-03-25T19:23:41.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 10 monclient(hunting): _negotiate 2024-03-25T19:23:41.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:41.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:41.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:23:41.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.089+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc568142520 con 0x7fc5681430c0 2024-03-25T19:23:41.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1728567084 0 0) 0x7fc550003520 con 0x7fc56813f710 2024-03-25T19:23:41.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 >> v1:172.21.15.123:6790/0 conn(0x7fc5681430c0 legacy=0x7fc56814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 >> v1:172.21.15.123:6789/0 conn(0x7fc5680a98f0 legacy=0x7fc5680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 10 monclient: _finish_hunting 0 2024-03-25T19:23:41.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 1 monclient: found mon.noname-c 2024-03-25T19:23:41.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:41.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:23:41.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5680b44c0 con 0x7fc56813f710 2024-03-25T19:23:41.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 10 monclient: _finish_auth 0 2024-03-25T19:23:41.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 10 monclient: _check_auth_tickets 2024-03-25T19:23:41.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56cbb8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:41.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56fe1e640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:23:41.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.098+0000 7fc56fe1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:41.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.102+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc55c001650 con 0x7fc56813f710 2024-03-25T19:23:41.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.102+0000 7fc56cbb8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.102+0000 7fc56cbb8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:41.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.102+0000 7fc56cbb8640 10 monclient: dump: 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:41.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.102+0000 7fc56fe1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:41.110 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.110+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc55c0018a0 con 0x7fc56813f710 2024-03-25T19:23:41.110 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.110+0000 7fc56cbb8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:41.110 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.110+0000 7fc56cbb8640 1 -- 172.21.15.123:0/31719327 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc55c001c30 con 0x7fc56813f710 2024-03-25T19:23:41.110 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.110+0000 7fc56fe1e640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:41.110 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.110+0000 7fc56fe1e640 4 set_mon_vals no callback set 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.110+0000 7fc56cbb8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.110+0000 7fc56cbb8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.110+0000 7fc56cbb8640 10 monclient: dump: 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:41.111 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:41.113 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.111+0000 7fc56fe1e640 10 monclient: shutdown 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.111+0000 7fc56fe1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.111+0000 7fc56fe1e640 1 -- 172.21.15.123:0/31719327 >> v1:172.21.15.151:6789/0 conn(0x7fc56813f710 legacy=0x7fc5681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.111+0000 7fc56fe1e640 1 -- 172.21.15.123:0/31719327 shutdown_connections 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.111+0000 7fc56fe1e640 1 -- 172.21.15.123:0/31719327 >> 172.21.15.123:0/31719327 conn(0x7fc5680b7ea0 msgr2=0x7fc5680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.111+0000 7fc56fe1e640 1 -- 172.21.15.123:0/31719327 shutdown_connections 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.111+0000 7fc56fe1e640 1 -- 172.21.15.123:0/31719327 wait complete. 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.112+0000 7fc56fe1e640 10 monclient: build_initial_monmap 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.112+0000 7fc56fe1e640 10 monclient: monmap: 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:41.114 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.112+0000 7fc56fe1e640 1 librados: starting msgr at 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.112+0000 7fc56fe1e640 1 librados: starting objecter 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.112+0000 7fc56fe1e640 1 Processor -- start 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 1 -- start start 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 1 librados: setting wanted keys 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 1 librados: calling monclient init 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 10 monclient: init 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:23:41.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 10 monclient(hunting): picked mon.noname-c con 0x7fc5680a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:41.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 10 monclient(hunting): picked mon.noname-b con 0x7fc56813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:41.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5681430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:41.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc568130f60 con 0x7fc5681430c0 2024-03-25T19:23:41.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5681353e0 con 0x7fc56813f710 2024-03-25T19:23:41.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc568136ab0 con 0x7fc5680a98f0 2024-03-25T19:23:41.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:41.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.113+0000 7fc56fe1e640 10 monclient(hunting): authenticate will time out at 5658.473600s 2024-03-25T19:23:41.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.122+0000 7fc56d3b9640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fc56813f710 0x7fc568130140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:55324/0 (socket says 172.21.15.123:55324) 2024-03-25T19:23:41.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.122+0000 7fc56d3b9640 1 -- 172.21.15.123:0/1067721820 learned_addr learned my addr 172.21.15.123:0/1067721820 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:41.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.123+0000 7fc5667fc640 10 client.?.objecter ms_handle_connect 0x7fc56813f710 2024-03-25T19:23:41.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.123+0000 7fc5667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:41.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.132+0000 7fc5667fc640 10 client.?.objecter ms_handle_connect 0x7fc5681430c0 2024-03-25T19:23:41.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.132+0000 7fc5667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:41.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.134+0000 7fc5667fc640 10 client.?.objecter ms_handle_connect 0x7fc5680a98f0 2024-03-25T19:23:41.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.134+0000 7fc5667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:41.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.138+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579209287 0 0) 0x7fc5681353e0 con 0x7fc56813f710 2024-03-25T19:23:41.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.138+0000 7fc5667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:41.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.138+0000 7fc5667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:41.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.138+0000 7fc5667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:41.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.138+0000 7fc5667fc640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:23:41.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.138+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc554004eb0 con 0x7fc56813f710 2024-03-25T19:23:41.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.141+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896715653 0 0) 0x7fc568130f60 con 0x7fc5681430c0 2024-03-25T19:23:41.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.141+0000 7fc5667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:41.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.141+0000 7fc5667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:41.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.141+0000 7fc5667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:41.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.141+0000 7fc5667fc640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:23:41.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.141+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5681353e0 con 0x7fc5681430c0 2024-03-25T19:23:41.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.148+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667664940 0 0) 0x7fc568136ab0 con 0x7fc5680a98f0 2024-03-25T19:23:41.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.148+0000 7fc5667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:41.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.148+0000 7fc5667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:41.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.148+0000 7fc5667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:41.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.148+0000 7fc5667fc640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:23:41.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.148+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc568130f60 con 0x7fc5680a98f0 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.162+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 484043693 0 0) 0x7fc554004eb0 con 0x7fc56813f710 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.162+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc568136ab0 con 0x7fc56813f710 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.162+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc55c001c70 con 0x7fc56813f710 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.162+0000 7fc5667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.162+0000 7fc5667fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.162+0000 7fc5667fc640 10 monclient(hunting): dump: 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.162 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.163 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:41.163 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:41.163 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:41.163 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:41.163 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:41.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.162+0000 7fc5667fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:41.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.168+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2258489935 0 0) 0x7fc568130f60 con 0x7fc5680a98f0 2024-03-25T19:23:41.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.168+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc554004eb0 con 0x7fc5680a98f0 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.168+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc560001c30 con 0x7fc5680a98f0 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.168+0000 7fc5667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.169+0000 7fc5667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.169+0000 7fc5667fc640 10 monclient(hunting): dump: 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:41.169 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:41.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.175+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3536301192 0 0) 0x7fc5681353e0 con 0x7fc5681430c0 2024-03-25T19:23:41.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.175+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc568130f60 con 0x7fc5681430c0 2024-03-25T19:23:41.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.175+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc5580017d0 con 0x7fc5681430c0 2024-03-25T19:23:41.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.175+0000 7fc5667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.175+0000 7fc5667fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:41.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.175+0000 7fc5667fc640 10 monclient(hunting): dump: 2024-03-25T19:23:41.176 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:41.176 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:41.176 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.176 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.176 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.176 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:41.176 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:41.176 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:41.176 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:41.176 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.178+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 447981746 0 0) 0x7fc554004eb0 con 0x7fc5680a98f0 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.178+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 >> v1:172.21.15.151:6789/0 conn(0x7fc56813f710 legacy=0x7fc568130140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 >> v1:172.21.15.123:6789/0 conn(0x7fc5681430c0 legacy=0x7fc568130850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc5667fc640 10 monclient: _finish_hunting 0 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc5667fc640 1 monclient: found mon.c 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc5667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc5667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc568136d90 con 0x7fc5680a98f0 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc5667fc640 10 monclient: _finish_auth 0 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc5667fc640 10 monclient: _check_auth_tickets 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc5667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc56fe1e640 5 monclient: authenticate success, global_id 4289 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc56fe1e640 10 monclient: _renew_subs 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc56fe1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc56fe1e640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc568137070 con 0x7fc5680a98f0 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc56fe1e640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc56fe1e640 10 monclient: _renew_subs 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc56fe1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc56fe1e640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5681380d0 con 0x7fc5680a98f0 2024-03-25T19:23:41.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.179+0000 7fc56fe1e640 1 librados: init done 2024-03-25T19:23:41.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.180+0000 7fc56fe1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:41.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.180+0000 7fc56fe1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:41.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.180+0000 7fc56fe1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:23:41.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.180+0000 7fc56fe1e640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5680627b0 con 0x7fc5680a98f0 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.497+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc560002060 con 0x7fc5680a98f0 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.497+0000 7fc5667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.497+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc560003b10 con 0x7fc5680a98f0 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.497+0000 7fc5667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.497+0000 7fc5667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.497+0000 7fc5667fc640 10 monclient: dump: 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:41.497 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.498 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:41.498 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:41.498 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:41.498 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:41.498 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:41.498 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.497+0000 7fc567fff640 4 set_mon_vals no callback set 2024-03-25T19:23:41.498 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.498+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fc560019b20 con 0x7fc5680a98f0 2024-03-25T19:23:41.498 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.498+0000 7fc56d3b9640 10 monclient: get_auth_request con 0x7fc55406e110 auth_method 0 2024-03-25T19:23:41.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.498+0000 7fc5667fc640 10 client.4289.objecter ms_handle_connect 0x7fc55406e110 2024-03-25T19:23:41.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.504+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fc5600538b0 con 0x7fc5680a98f0 2024-03-25T19:23:41.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.504+0000 7fc5667fc640 10 client.4289.objecter ms_dispatch 0x7fc568000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:41.505 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.504+0000 7fc5667fc640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:41.505 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.504+0000 7fc5667fc640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:41.505 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.504+0000 7fc5667fc640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-25T19:23:41.505 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.504+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc56004f1f0 con 0x7fc5680a98f0 2024-03-25T19:23:41.505 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.504+0000 7fc5667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:41.505 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.504+0000 7fc5667fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:41.763 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.763+0000 7fc56fe1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:41.763 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.763+0000 7fc56fe1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:41.763 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.763+0000 7fc56fe1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:23:41.763 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.763+0000 7fc56fe1e640 1 -- 172.21.15.123:0/1067721820 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5680aae40 con 0x7fc5680a98f0 2024-03-25T19:23:41.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.834+0000 7fc5667fc640 1 -- 172.21.15.123:0/1067721820 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fc5600512f0 con 0x7fc5680a98f0 2024-03-25T19:23:41.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.834+0000 7fc5667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:41.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.834+0000 7fc5667fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:41.835 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:23:41.835 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 10 librados: watch_flush enter 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 10 librados: watch_flush exit 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 10 client.4289.objecter successfully canceled tick 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 1 -- 172.21.15.123:0/1067721820 >> v1:172.21.15.151:6800/727684239 conn(0x7fc55406e110 legacy=0x7fc5540705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 10 monclient: shutdown 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 1 -- 172.21.15.123:0/1067721820 >> v1:172.21.15.123:6790/0 conn(0x7fc5680a98f0 legacy=0x7fc5681377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 1 -- 172.21.15.123:0/1067721820 shutdown_connections 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 1 -- 172.21.15.123:0/1067721820 >> 172.21.15.123:0/1067721820 conn(0x7fc5680b7ea0 msgr2=0x7fc568074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:41.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 1 -- 172.21.15.123:0/1067721820 shutdown_connections 2024-03-25T19:23:41.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 1 -- 172.21.15.123:0/1067721820 wait complete. 2024-03-25T19:23:41.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:41.837+0000 7fc56fe1e640 1 librados: shutdown 2024-03-25T19:23:41.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:48.656 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:48.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.755+0000 7fa07cfd6640 10 monclient: get_monmap_and_config 2024-03-25T19:23:48.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.755+0000 7fa07cfd6640 10 monclient: build_initial_monmap 2024-03-25T19:23:48.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.755+0000 7fa07cfd6640 10 monclient: monmap: 2024-03-25T19:23:48.755 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:48.755 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:48.755 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:23:48.756490+0000 2024-03-25T19:23:48.755 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:23:48.756490+0000 2024-03-25T19:23:48.755 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:48.756 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:48.756 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:48.756 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:23:48.756 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:23:48.756 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:48.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.756+0000 7fa07cfd6640 1 Processor -- start 2024-03-25T19:23:48.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.756+0000 7fa07cfd6640 1 -- start start 2024-03-25T19:23:48.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.756+0000 7fa07cfd6640 10 monclient: init 2024-03-25T19:23:48.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.756+0000 7fa07cfd6640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:48.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.756+0000 7fa07cfd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:48.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.757+0000 7fa07cfd6640 10 monclient(hunting): picked mon.noname-c con 0x7fa0780b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:48.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.757+0000 7fa07cfd6640 10 monclient(hunting): picked mon.noname-b con 0x7fa0780b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:48.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.757+0000 7fa07cfd6640 10 monclient(hunting): picked mon.noname-a con 0x7fa07814be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:48.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.757+0000 7fa07cfd6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0780af940 con 0x7fa07814be90 2024-03-25T19:23:48.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.757+0000 7fa07cfd6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0780b0a70 con 0x7fa0780b4d60 2024-03-25T19:23:48.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.757+0000 7fa07cfd6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0780b0d50 con 0x7fa0780b1730 2024-03-25T19:23:48.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.757+0000 7fa07cfd6640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:48.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.757+0000 7fa07cfd6640 10 monclient(hunting): authenticate will time out at 5666.116992s 2024-03-25T19:23:48.763 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.763+0000 7fa076d9d640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa07814be90 0x7fa07814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:38718/0 (socket says 172.21.15.123:38718) 2024-03-25T19:23:48.763 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.763+0000 7fa076d9d640 1 -- 172.21.15.123:0/2591128672 learned_addr learned my addr 172.21.15.123:0/2591128672 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:48.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.782+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986922757 0 0) 0x7fa0780af940 con 0x7fa07814be90 2024-03-25T19:23:48.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.782+0000 7fa075d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.782+0000 7fa075d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.782+0000 7fa075d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.782+0000 7fa075d9b640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:23:48.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.782+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa064003520 con 0x7fa07814be90 2024-03-25T19:23:48.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.788+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348199287 0 0) 0x7fa0780b0a70 con 0x7fa0780b4d60 2024-03-25T19:23:48.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.788+0000 7fa075d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.788+0000 7fa075d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.788+0000 7fa075d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.788+0000 7fa075d9b640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:23:48.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.788+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0780af940 con 0x7fa0780b4d60 2024-03-25T19:23:48.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.789+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245505996 0 0) 0x7fa0780b0d50 con 0x7fa0780b1730 2024-03-25T19:23:48.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.789+0000 7fa075d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.789+0000 7fa075d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.789+0000 7fa075d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.789+0000 7fa075d9b640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:23:48.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.789+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0780b0a70 con 0x7fa0780b1730 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 583968953 0 0) 0x7fa0780af940 con 0x7fa0780b4d60 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 >> v1:172.21.15.123:6790/0 conn(0x7fa0780b1730 legacy=0x7fa0780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 >> v1:172.21.15.123:6789/0 conn(0x7fa07814be90 legacy=0x7fa07814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 1 monclient: found mon.noname-c 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0780b1030 con 0x7fa0780b4d60 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 10 monclient: _finish_auth 0 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa068001670 con 0x7fa0780b4d60 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa075d9b640 10 monclient: dump: 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:48.800 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa07cfd6640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.799+0000 7fa07cfd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:48.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.807+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0680017d0 con 0x7fa0780b4d60 2024-03-25T19:23:48.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.807+0000 7fa075d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:48.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.807+0000 7fa075d9b640 1 -- 172.21.15.123:0/2591128672 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa068001be0 con 0x7fa0780b4d60 2024-03-25T19:23:48.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.807+0000 7fa07cfd6640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:48.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.807+0000 7fa07cfd6640 4 set_mon_vals no callback set 2024-03-25T19:23:48.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.807+0000 7fa075d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.807+0000 7fa075d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.807+0000 7fa075d9b640 10 monclient: dump: 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:48.808 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:48.809 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.808+0000 7fa07cfd6640 10 monclient: shutdown 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.808+0000 7fa07cfd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.808+0000 7fa07cfd6640 1 -- 172.21.15.123:0/2591128672 >> v1:172.21.15.151:6789/0 conn(0x7fa0780b4d60 legacy=0x7fa07813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.808+0000 7fa07cfd6640 1 -- 172.21.15.123:0/2591128672 shutdown_connections 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.808+0000 7fa07cfd6640 1 -- 172.21.15.123:0/2591128672 >> 172.21.15.123:0/2591128672 conn(0x7fa0780617a0 msgr2=0x7fa078061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.808+0000 7fa07cfd6640 1 -- 172.21.15.123:0/2591128672 shutdown_connections 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.808+0000 7fa07cfd6640 1 -- 172.21.15.123:0/2591128672 wait complete. 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.809+0000 7fa07cfd6640 10 monclient: build_initial_monmap 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.809+0000 7fa07cfd6640 10 monclient: monmap: 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:23:48.810 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 1 librados: starting msgr at 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 1 librados: starting objecter 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 1 Processor -- start 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 1 -- start start 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 1 librados: setting wanted keys 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 1 librados: calling monclient init 2024-03-25T19:23:48.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 10 monclient: init 2024-03-25T19:23:48.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:48.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:48.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 10 monclient(hunting): picked mon.noname-a con 0x7fa0780b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:48.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 10 monclient(hunting): picked mon.noname-c con 0x7fa0780b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:48.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 10 monclient(hunting): picked mon.noname-b con 0x7fa07814be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:48.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa078137750 con 0x7fa0780b1730 2024-03-25T19:23:48.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0781dbe40 con 0x7fa07814be90 2024-03-25T19:23:48.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0781379c0 con 0x7fa0780b4d60 2024-03-25T19:23:48.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:48.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.810+0000 7fa07cfd6640 10 monclient(hunting): authenticate will time out at 5666.170761s 2024-03-25T19:23:48.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.824+0000 7fa06ffff640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fa0780b4d60 0x7fa0781db020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:35658/0 (socket says 172.21.15.123:35658) 2024-03-25T19:23:48.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.824+0000 7fa06ffff640 1 -- 172.21.15.123:0/1637673262 learned_addr learned my addr 172.21.15.123:0/1637673262 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:48.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.825+0000 7fa06effd640 10 client.?.objecter ms_handle_connect 0x7fa0780b4d60 2024-03-25T19:23:48.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.825+0000 7fa06effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:48.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.829+0000 7fa06effd640 10 client.?.objecter ms_handle_connect 0x7fa0780b1730 2024-03-25T19:23:48.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.829+0000 7fa06effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:48.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.835+0000 7fa06effd640 10 client.?.objecter ms_handle_connect 0x7fa07814be90 2024-03-25T19:23:48.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.835+0000 7fa06effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:48.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.835+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456732999 0 0) 0x7fa0781379c0 con 0x7fa0780b4d60 2024-03-25T19:23:48.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.835+0000 7fa06effd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.835+0000 7fa06effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.835+0000 7fa06effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.835+0000 7fa06effd640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:23:48.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.835+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa058004eb0 con 0x7fa0780b4d60 2024-03-25T19:23:48.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.837+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093075514 0 0) 0x7fa078137750 con 0x7fa0780b1730 2024-03-25T19:23:48.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.837+0000 7fa06effd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.837+0000 7fa06effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.837+0000 7fa06effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.837+0000 7fa06effd640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:23:48.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.837+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0781379c0 con 0x7fa0780b1730 2024-03-25T19:23:48.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.841+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609825026 0 0) 0x7fa0781dbe40 con 0x7fa07814be90 2024-03-25T19:23:48.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.841+0000 7fa06effd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.841+0000 7fa06effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.841+0000 7fa06effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.841+0000 7fa06effd640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:23:48.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.841+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa078137750 con 0x7fa07814be90 2024-03-25T19:23:48.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.855+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1201988520 0 0) 0x7fa0781379c0 con 0x7fa0780b1730 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.856+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0781dbe40 con 0x7fa0780b1730 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.856+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa068001ae0 con 0x7fa0780b1730 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.856+0000 7fa06effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.856+0000 7fa06effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.856+0000 7fa06effd640 10 monclient(hunting): dump: 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:48.857 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:48.857 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:48.857 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:48.857 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.856+0000 7fa06effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.862+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2988178334 0 0) 0x7fa078137750 con 0x7fa07814be90 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.862+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0781379c0 con 0x7fa07814be90 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.862+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa070001a30 con 0x7fa07814be90 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.862+0000 7fa06effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.862+0000 7fa06effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.862+0000 7fa06effd640 10 monclient(hunting): dump: 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:48.862 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:48.863 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:48.863 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:48.863 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:48.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.866+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 48964396 0 0) 0x7fa058004eb0 con 0x7fa0780b4d60 2024-03-25T19:23:48.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.866+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa078137750 con 0x7fa0780b4d60 2024-03-25T19:23:48.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.866+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa060001a10 con 0x7fa0780b4d60 2024-03-25T19:23:48.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.866+0000 7fa06effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:48.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.866+0000 7fa06effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:48.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.866+0000 7fa06effd640 10 monclient(hunting): dump: 2024-03-25T19:23:48.866 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:48.866 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:48.866 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:48.866 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:48.867 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:48.867 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:48.867 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:48.867 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:48.867 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:48.867 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:48.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3488690651 0 0) 0x7fa0781379c0 con 0x7fa07814be90 2024-03-25T19:23:48.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 >> v1:172.21.15.123:6790/0 conn(0x7fa0780b4d60 legacy=0x7fa0781db020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:48.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 >> v1:172.21.15.123:6789/0 conn(0x7fa0780b1730 legacy=0x7fa0781405c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:48.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 1 monclient: found mon.b 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa078140cd0 con 0x7fa07814be90 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 10 monclient: _finish_auth 0 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa06effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa07cfd6640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa07cfd6640 10 monclient: _renew_subs 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa07cfd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa07cfd6640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0781431c0 con 0x7fa07814be90 2024-03-25T19:23:48.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa07cfd6640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:48.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa07cfd6640 10 monclient: _renew_subs 2024-03-25T19:23:48.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa07cfd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:48.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa07cfd6640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa078135ce0 con 0x7fa07814be90 2024-03-25T19:23:48.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.878+0000 7fa07cfd6640 1 librados: init done 2024-03-25T19:23:48.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.879+0000 7fa07cfd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:48.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.879+0000 7fa07cfd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.879+0000 7fa07cfd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:48.879+0000 7fa07cfd6640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa078135ee0 con 0x7fa07814be90 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.166+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa070001f20 con 0x7fa07814be90 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.166+0000 7fa06effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.166+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa070004b60 con 0x7fa07814be90 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.166+0000 7fa06effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.166+0000 7fa06effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.166+0000 7fa06effd640 10 monclient: dump: 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:49.166 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:49.167 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:49.167 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:49.167 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:49.167 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:49.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.166+0000 7fa07559a640 4 set_mon_vals no callback set 2024-03-25T19:23:49.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.167+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fa07001ab30 con 0x7fa07814be90 2024-03-25T19:23:49.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.167+0000 7fa07659c640 10 monclient: get_auth_request con 0x7fa05806e160 auth_method 0 2024-03-25T19:23:49.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.168+0000 7fa06effd640 10 client.4288.objecter ms_handle_connect 0x7fa05806e160 2024-03-25T19:23:49.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.172+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fa0700546b0 con 0x7fa07814be90 2024-03-25T19:23:49.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.172+0000 7fa06effd640 10 client.4288.objecter ms_dispatch 0x7fa078000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:49.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.172+0000 7fa06effd640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:49.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.172+0000 7fa06effd640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:49.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.172+0000 7fa06effd640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:23:49.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.172+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0700501c0 con 0x7fa07814be90 2024-03-25T19:23:49.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.172+0000 7fa06effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:49.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.172+0000 7fa06effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:49.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.420+0000 7fa07cfd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:49.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.420+0000 7fa07cfd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:49.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.420+0000 7fa07cfd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:23:49.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.420+0000 7fa07cfd6640 1 -- 172.21.15.123:0/1637673262 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0780627b0 con 0x7fa07814be90 2024-03-25T19:23:49.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.459+0000 7fa06effd640 1 -- 172.21.15.123:0/1637673262 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fa0700520f0 con 0x7fa07814be90 2024-03-25T19:23:49.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.459+0000 7fa06effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:49.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.459+0000 7fa06effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:49.459 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:23:49.459 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:49.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.461+0000 7fa07cfd6640 10 librados: watch_flush enter 2024-03-25T19:23:49.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.461+0000 7fa07cfd6640 10 librados: watch_flush exit 2024-03-25T19:23:49.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.461+0000 7fa07cfd6640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:23:49.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.461+0000 7fa07cfd6640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:23:49.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.462+0000 7fa07cfd6640 1 -- 172.21.15.123:0/1637673262 >> v1:172.21.15.151:6800/727684239 conn(0x7fa05806e160 legacy=0x7fa058070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:49.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.462+0000 7fa07cfd6640 10 monclient: shutdown 2024-03-25T19:23:49.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.462+0000 7fa07cfd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:49.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.462+0000 7fa07cfd6640 1 -- 172.21.15.123:0/1637673262 >> v1:172.21.15.151:6789/0 conn(0x7fa07814be90 legacy=0x7fa0781db730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:49.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.462+0000 7fa07cfd6640 1 -- 172.21.15.123:0/1637673262 shutdown_connections 2024-03-25T19:23:49.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.462+0000 7fa07cfd6640 1 -- 172.21.15.123:0/1637673262 >> 172.21.15.123:0/1637673262 conn(0x7fa0780617a0 msgr2=0x7fa0780b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:49.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.462+0000 7fa07cfd6640 1 -- 172.21.15.123:0/1637673262 shutdown_connections 2024-03-25T19:23:49.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.462+0000 7fa07cfd6640 1 -- 172.21.15.123:0/1637673262 wait complete. 2024-03-25T19:23:49.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:49.462+0000 7fa07cfd6640 1 librados: shutdown 2024-03-25T19:23:49.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:49.635 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:01:39.207 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.688 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:01:39.260 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.737 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:01:39.271 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.739 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:01:39.311 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.742 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:01:39.276 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.746 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:01:39.319 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.752 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:01:39.324 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.797 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:01:39.330 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:49.801 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:01:39.374 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:56.280 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:56.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.395+0000 7fe50a93b640 10 monclient: get_monmap_and_config 2024-03-25T19:23:56.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.395+0000 7fe50a93b640 10 monclient: build_initial_monmap 2024-03-25T19:23:56.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.395+0000 7fe50a93b640 10 monclient: monmap: 2024-03-25T19:23:56.395 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:56.395 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:56.395 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:23:56.396279+0000 2024-03-25T19:23:56.395 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:23:56.396279+0000 2024-03-25T19:23:56.395 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:56.396 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:56.396 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:56.396 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:23:56.396 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:23:56.396 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 1 Processor -- start 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 1 -- start start 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 10 monclient: init 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 10 monclient(hunting): picked mon.noname-c con 0x7fe5040a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 10 monclient(hunting): picked mon.noname-a con 0x7fe50413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 10 monclient(hunting): picked mon.noname-b con 0x7fe5041430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe504142520 con 0x7fe50413f710 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5040b28a0 con 0x7fe5040a98f0 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5040b2a30 con 0x7fe5041430c0 2024-03-25T19:23:56.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:56.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.396+0000 7fe50a93b640 10 monclient(hunting): authenticate will time out at 5673.756804s 2024-03-25T19:23:56.399 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.399+0000 7fe5037fe640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe50413f710 0x7fe5041429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:34074/0 (socket says 172.21.15.123:34074) 2024-03-25T19:23:56.399 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.399+0000 7fe5037fe640 1 -- 172.21.15.123:0/1281857997 learned_addr learned my addr 172.21.15.123:0/1281857997 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:56.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.403+0000 7fe502ffd640 1 -- 172.21.15.123:0/1281857997 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104877755 0 0) 0x7fe504142520 con 0x7fe50413f710 2024-03-25T19:23:56.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.403+0000 7fe502ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:56.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.403+0000 7fe502ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:56.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.403+0000 7fe502ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:56.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.403+0000 7fe502ffd640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:23:56.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.403+0000 7fe502ffd640 1 -- 172.21.15.123:0/1281857997 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ec003520 con 0x7fe50413f710 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.416+0000 7fe502ffd640 1 -- 172.21.15.123:0/1281857997 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1923750861 0 0) 0x7fe4ec003520 con 0x7fe50413f710 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 1 -- 172.21.15.123:0/1281857997 >> v1:172.21.15.123:6790/0 conn(0x7fe5041430c0 legacy=0x7fe50414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 1 -- 172.21.15.123:0/1281857997 >> v1:172.21.15.151:6789/0 conn(0x7fe5040a98f0 legacy=0x7fe5040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 1 monclient: found mon.noname-a 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 1 -- 172.21.15.123:0/1281857997 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5040b44c0 con 0x7fe50413f710 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 10 monclient: _finish_auth 0 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 1 -- 172.21.15.123:0/1281857997 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe4f8001690 con 0x7fe50413f710 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe502ffd640 10 monclient: dump: 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:56.418 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:56.419 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:56.419 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:56.419 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:56.419 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:56.419 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe50a93b640 5 monclient: authenticate success, global_id 4272 2024-03-25T19:23:56.419 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.417+0000 7fe50a93b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:56.431 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.431+0000 7fe502ffd640 1 -- 172.21.15.123:0/1281857997 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4f80017f0 con 0x7fe50413f710 2024-03-25T19:23:56.431 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.431+0000 7fe502ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:56.431 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.431+0000 7fe502ffd640 1 -- 172.21.15.123:0/1281857997 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe4f8001c00 con 0x7fe50413f710 2024-03-25T19:23:56.431 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.431+0000 7fe502ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:56.431 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.431+0000 7fe502ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:56.431 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.431+0000 7fe502ffd640 10 monclient: dump: 2024-03-25T19:23:56.431 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:56.431 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:56.431 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:56.431 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:56.432 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:56.432 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:56.432 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:56.432 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:56.432 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:56.432 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:56.432 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.431+0000 7fe50a93b640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:56.432 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.431+0000 7fe50a93b640 4 set_mon_vals no callback set 2024-03-25T19:23:56.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.432+0000 7fe50a93b640 10 monclient: shutdown 2024-03-25T19:23:56.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.432+0000 7fe50a93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:56.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.432+0000 7fe50a93b640 1 -- 172.21.15.123:0/1281857997 >> v1:172.21.15.123:6789/0 conn(0x7fe50413f710 legacy=0x7fe5041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:56.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.432+0000 7fe50a93b640 1 -- 172.21.15.123:0/1281857997 shutdown_connections 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.432+0000 7fe50a93b640 1 -- 172.21.15.123:0/1281857997 >> 172.21.15.123:0/1281857997 conn(0x7fe5040b7ea0 msgr2=0x7fe5040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.432+0000 7fe50a93b640 1 -- 172.21.15.123:0/1281857997 shutdown_connections 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.432+0000 7fe50a93b640 1 -- 172.21.15.123:0/1281857997 wait complete. 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.433+0000 7fe50a93b640 10 monclient: build_initial_monmap 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.433+0000 7fe50a93b640 10 monclient: monmap: 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:23:56.434 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:23:56.435 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:23:56.435 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:56.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.433+0000 7fe50a93b640 1 librados: starting msgr at 2024-03-25T19:23:56.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.433+0000 7fe50a93b640 1 librados: starting objecter 2024-03-25T19:23:56.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.433+0000 7fe50a93b640 1 Processor -- start 2024-03-25T19:23:56.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.433+0000 7fe50a93b640 1 -- start start 2024-03-25T19:23:56.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.433+0000 7fe50a93b640 1 librados: setting wanted keys 2024-03-25T19:23:56.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.433+0000 7fe50a93b640 1 librados: calling monclient init 2024-03-25T19:23:56.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.433+0000 7fe50a93b640 10 monclient: init 2024-03-25T19:23:56.436 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.434+0000 7fe50a93b640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:56.436 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.434+0000 7fe50a93b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:23:56.436 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.434+0000 7fe50a93b640 10 monclient(hunting): picked mon.noname-c con 0x7fe5040a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:23:56.436 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.434+0000 7fe50a93b640 10 monclient(hunting): picked mon.noname-b con 0x7fe50413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:23:56.436 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.434+0000 7fe50a93b640 10 monclient(hunting): picked mon.noname-a con 0x7fe5041430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:23:56.436 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.434+0000 7fe50a93b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5040abd70 con 0x7fe5041430c0 2024-03-25T19:23:56.436 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.434+0000 7fe50a93b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5041d7710 con 0x7fe50413f710 2024-03-25T19:23:56.436 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.434+0000 7fe50a93b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5041d8340 con 0x7fe5040a98f0 2024-03-25T19:23:56.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.434+0000 7fe50a93b640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:56.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.434+0000 7fe50a93b640 10 monclient(hunting): authenticate will time out at 5673.794552s 2024-03-25T19:23:56.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.440+0000 7fe508ed8640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe5041430c0 0x7fe5041d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:34104/0 (socket says 172.21.15.123:34104) 2024-03-25T19:23:56.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.440+0000 7fe508ed8640 1 -- 172.21.15.123:0/3311908731 learned_addr learned my addr 172.21.15.123:0/3311908731 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:23:56.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.443+0000 7fe500ff9640 10 client.?.objecter ms_handle_connect 0x7fe5041430c0 2024-03-25T19:23:56.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.443+0000 7fe500ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:56.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.443+0000 7fe500ff9640 10 client.?.objecter ms_handle_connect 0x7fe50413f710 2024-03-25T19:23:56.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.443+0000 7fe500ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:56.445 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.445+0000 7fe500ff9640 10 client.?.objecter ms_handle_connect 0x7fe5040a98f0 2024-03-25T19:23:56.445 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.446+0000 7fe500ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:56.448 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.448+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089973322 0 0) 0x7fe5040abd70 con 0x7fe5041430c0 2024-03-25T19:23:56.448 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.448+0000 7fe500ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:56.448 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.448+0000 7fe500ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:56.449 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.448+0000 7fe500ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:56.449 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.448+0000 7fe500ff9640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:23:56.449 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.449+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f0004eb0 con 0x7fe5041430c0 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.459+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531146012 0 0) 0x7fe5041d7710 con 0x7fe50413f710 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.459+0000 7fe500ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.459+0000 7fe500ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.459+0000 7fe500ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.459+0000 7fe500ff9640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.459+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5040abd70 con 0x7fe50413f710 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.459+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462013734 0 0) 0x7fe5041d8340 con 0x7fe5040a98f0 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.460+0000 7fe500ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.460+0000 7fe500ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.460+0000 7fe500ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.460+0000 7fe500ff9640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.460+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5041d7710 con 0x7fe5040a98f0 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.461+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3282743504 0 0) 0x7fe4f0004eb0 con 0x7fe5041430c0 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.461+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5041d8340 con 0x7fe5041430c0 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.461+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe4fc001c90 con 0x7fe5041430c0 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.461+0000 7fe500ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.461+0000 7fe500ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.461+0000 7fe500ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:56.462 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:56.462 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:56.462 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:56.462 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:56.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.461+0000 7fe500ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2287015176 0 0) 0x7fe5041d8340 con 0x7fe5041430c0 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 >> v1:172.21.15.123:6790/0 conn(0x7fe5040a98f0 legacy=0x7fe504137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 >> v1:172.21.15.151:6789/0 conn(0x7fe50413f710 legacy=0x7fe5041d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 1 monclient: found mon.a 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5041d84d0 con 0x7fe5041430c0 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 10 monclient: _finish_auth 0 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe500ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe50a93b640 5 monclient: authenticate success, global_id 4275 2024-03-25T19:23:56.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe50a93b640 10 monclient: _renew_subs 2024-03-25T19:23:56.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe50a93b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:23:56.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe50a93b640 1 -- 172.21.15.123:0/3311908731 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5041d41e0 con 0x7fe5041430c0 2024-03-25T19:23:56.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe50a93b640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:56.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe50a93b640 10 monclient: _renew_subs 2024-03-25T19:23:56.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe50a93b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:23:56.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe50a93b640 1 -- 172.21.15.123:0/3311908731 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5041d5150 con 0x7fe5041430c0 2024-03-25T19:23:56.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.468+0000 7fe50a93b640 1 librados: init done 2024-03-25T19:23:56.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.469+0000 7fe50a93b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:56.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.469+0000 7fe50a93b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:56.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.469+0000 7fe50a93b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:23:56.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.469+0000 7fe50a93b640 1 -- 172.21.15.123:0/3311908731 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5040627b0 con 0x7fe5041430c0 2024-03-25T19:23:56.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.760+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4fc003990 con 0x7fe5041430c0 2024-03-25T19:23:56.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.760+0000 7fe500ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:56.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.760+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe4fc003da0 con 0x7fe5041430c0 2024-03-25T19:23:56.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.760+0000 7fe500ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:56.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.760+0000 7fe500ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.760+0000 7fe500ff9640 10 monclient: dump: 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:23:56.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.760+0000 7fe5027fc640 4 set_mon_vals no callback set 2024-03-25T19:23:56.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.761+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fe4fc003fc0 con 0x7fe5041430c0 2024-03-25T19:23:56.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.762+0000 7fe503fff640 10 monclient: get_auth_request con 0x7fe4f006de20 auth_method 0 2024-03-25T19:23:56.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.762+0000 7fe500ff9640 10 client.4275.objecter ms_handle_connect 0x7fe4f006de20 2024-03-25T19:23:56.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.766+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fe4fc054b20 con 0x7fe5041430c0 2024-03-25T19:23:56.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.766+0000 7fe500ff9640 10 client.4275.objecter ms_dispatch 0x7fe504000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:56.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.766+0000 7fe500ff9640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:56.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.766+0000 7fe500ff9640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:56.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.767+0000 7fe500ff9640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-25T19:23:56.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.767+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4fc050630 con 0x7fe5041430c0 2024-03-25T19:23:56.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.767+0000 7fe500ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:56.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.767+0000 7fe500ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:56.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.981+0000 7fe50a93b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:56.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.981+0000 7fe50a93b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:56.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.981+0000 7fe50a93b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:23:56.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:56.981+0000 7fe50a93b640 1 -- 172.21.15.123:0/3311908731 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5041d78a0 con 0x7fe5041430c0 2024-03-25T19:23:57.004 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.005+0000 7fe500ff9640 1 -- 172.21.15.123:0/3311908731 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fe4fc052560 con 0x7fe5041430c0 2024-03-25T19:23:57.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.005+0000 7fe500ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:57.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.005+0000 7fe500ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:57.005 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:23:57.005 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:57.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 10 librados: watch_flush enter 2024-03-25T19:23:57.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 10 librados: watch_flush exit 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 10 client.4275.objecter successfully canceled tick 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 1 -- 172.21.15.123:0/3311908731 >> v1:172.21.15.151:6800/727684239 conn(0x7fe4f006de20 legacy=0x7fe4f00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 10 monclient: shutdown 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 1 -- 172.21.15.123:0/3311908731 >> v1:172.21.15.123:6789/0 conn(0x7fe5041430c0 legacy=0x7fe5041d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 1 -- 172.21.15.123:0/3311908731 shutdown_connections 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 1 -- 172.21.15.123:0/3311908731 >> 172.21.15.123:0/3311908731 conn(0x7fe5040b7ea0 msgr2=0x7fe504074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 1 -- 172.21.15.123:0/3311908731 shutdown_connections 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 1 -- 172.21.15.123:0/3311908731 wait complete. 2024-03-25T19:23:57.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:23:57.007+0000 7fe50a93b640 1 librados: shutdown 2024-03-25T19:23:57.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:03.825 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:03.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.939+0000 7f948468d640 10 monclient: get_monmap_and_config 2024-03-25T19:24:03.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.939+0000 7f948468d640 10 monclient: build_initial_monmap 2024-03-25T19:24:03.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.939+0000 7f948468d640 10 monclient: monmap: 2024-03-25T19:24:03.940 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:03.941 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:03.941 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:24:03.941430+0000 2024-03-25T19:24:03.941 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:24:03.941430+0000 2024-03-25T19:24:03.941 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:03.941 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:03.941 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:03.941 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:24:03.941 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:24:03.941 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:03.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.940+0000 7f948468d640 1 Processor -- start 2024-03-25T19:24:03.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.940+0000 7f948468d640 1 -- start start 2024-03-25T19:24:03.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.940+0000 7f948468d640 10 monclient: init 2024-03-25T19:24:03.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.940+0000 7f948468d640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:03.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.941+0000 7f948468d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:24:03.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.941+0000 7f948468d640 10 monclient(hunting): picked mon.noname-a con 0x7f947c0b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:03.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.941+0000 7f948468d640 10 monclient(hunting): picked mon.noname-b con 0x7f947c0b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:03.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.941+0000 7f948468d640 10 monclient(hunting): picked mon.noname-c con 0x7f947c14be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:03.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.941+0000 7f948468d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947c0af940 con 0x7f947c0b4d60 2024-03-25T19:24:03.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.941+0000 7f948468d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947c0b0a70 con 0x7f947c14be90 2024-03-25T19:24:03.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.941+0000 7f948468d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947c0b0d50 con 0x7f947c0b1730 2024-03-25T19:24:03.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.941+0000 7f948468d640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:03.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.941+0000 7f948468d640 10 monclient(hunting): authenticate will time out at 5681.302096s 2024-03-25T19:24:03.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.952+0000 7f9482429640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f947c0b4d60 0x7f947c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:38730/0 (socket says 172.21.15.123:38730) 2024-03-25T19:24:03.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.952+0000 7f9482429640 1 -- 172.21.15.123:0/1788061843 learned_addr learned my addr 172.21.15.123:0/1788061843 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:03.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.963+0000 7f9481427640 1 -- 172.21.15.123:0/1788061843 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637812507 0 0) 0x7f947c0b0d50 con 0x7f947c0b1730 2024-03-25T19:24:03.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.963+0000 7f9481427640 10 monclient(hunting): _negotiate 2024-03-25T19:24:03.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.963+0000 7f9481427640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:03.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.963+0000 7f9481427640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:03.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.963+0000 7f9481427640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:24:03.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.963+0000 7f9481427640 1 -- 172.21.15.123:0/1788061843 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9468003520 con 0x7f947c0b1730 2024-03-25T19:24:03.969 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 1 -- 172.21.15.123:0/1788061843 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 495969940 0 0) 0x7f9468003520 con 0x7f947c0b1730 2024-03-25T19:24:03.969 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 1 -- 172.21.15.123:0/1788061843 >> v1:172.21.15.151:6789/0 conn(0x7f947c14be90 legacy=0x7f947c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:03.969 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 1 -- 172.21.15.123:0/1788061843 >> v1:172.21.15.123:6789/0 conn(0x7f947c0b4d60 legacy=0x7f947c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 10 monclient: _finish_hunting 0 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 1 monclient: found mon.noname-b 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 1 -- 172.21.15.123:0/1788061843 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c0b1030 con 0x7f947c0b1730 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 10 monclient: _finish_auth 0 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 10 monclient: _check_auth_tickets 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 1 -- 172.21.15.123:0/1788061843 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f946c001690 con 0x7f947c0b1730 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:03.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f9481427640 10 monclient: dump: 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f948468d640 5 monclient: authenticate success, global_id 4304 2024-03-25T19:24:03.971 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.968+0000 7f948468d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:03.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.975+0000 7f9481427640 1 -- 172.21.15.123:0/1788061843 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f946c001830 con 0x7f947c0b1730 2024-03-25T19:24:03.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.975+0000 7f9481427640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:03.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.975+0000 7f948468d640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:03.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.975+0000 7f948468d640 4 set_mon_vals no callback set 2024-03-25T19:24:03.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.975+0000 7f9481427640 1 -- 172.21.15.123:0/1788061843 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f946c001c00 con 0x7f947c0b1730 2024-03-25T19:24:03.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.975+0000 7f9481427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.975+0000 7f9481427640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.975+0000 7f9481427640 10 monclient: dump: 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:03.977 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:03.979 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.976+0000 7f948468d640 10 monclient: shutdown 2024-03-25T19:24:03.979 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.976+0000 7f948468d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:03.979 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.976+0000 7f948468d640 1 -- 172.21.15.123:0/1788061843 >> v1:172.21.15.123:6790/0 conn(0x7f947c0b1730 legacy=0x7f947c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:03.979 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.976+0000 7f948468d640 1 -- 172.21.15.123:0/1788061843 shutdown_connections 2024-03-25T19:24:03.979 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.976+0000 7f948468d640 1 -- 172.21.15.123:0/1788061843 >> 172.21.15.123:0/1788061843 conn(0x7f947c0617a0 msgr2=0x7f947c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:03.979 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.977+0000 7f948468d640 1 -- 172.21.15.123:0/1788061843 shutdown_connections 2024-03-25T19:24:03.979 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.977+0000 7f948468d640 1 -- 172.21.15.123:0/1788061843 wait complete. 2024-03-25T19:24:03.979 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.977+0000 7f948468d640 10 monclient: build_initial_monmap 2024-03-25T19:24:03.979 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.977+0000 7f948468d640 10 monclient: monmap: 2024-03-25T19:24:03.979 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 1 librados: starting msgr at 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 1 librados: starting objecter 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 1 Processor -- start 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 1 -- start start 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 1 librados: setting wanted keys 2024-03-25T19:24:03.980 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 1 librados: calling monclient init 2024-03-25T19:24:03.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 10 monclient: init 2024-03-25T19:24:03.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:03.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:03.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 10 monclient(hunting): picked mon.noname-b con 0x7f947c0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:03.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.978+0000 7f948468d640 10 monclient(hunting): picked mon.noname-c con 0x7f947c0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:03.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.979+0000 7f948468d640 10 monclient(hunting): picked mon.noname-a con 0x7f947c14be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:03.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.979+0000 7f948468d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947c137010 con 0x7f947c14be90 2024-03-25T19:24:03.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.979+0000 7f948468d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947c1362c0 con 0x7f947c0b1730 2024-03-25T19:24:03.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.979+0000 7f948468d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947c136450 con 0x7f947c0b4d60 2024-03-25T19:24:03.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.979+0000 7f948468d640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:03.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.979+0000 7f948468d640 10 monclient(hunting): authenticate will time out at 5681.340021s 2024-03-25T19:24:03.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.983+0000 7f9482429640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f947c0b1730 0x7f947c136900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:56224/0 (socket says 172.21.15.123:56224) 2024-03-25T19:24:03.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.983+0000 7f9482429640 1 -- 172.21.15.123:0/491891011 learned_addr learned my addr 172.21.15.123:0/491891011 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:03.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.983+0000 7f9472ffd640 10 client.?.objecter ms_handle_connect 0x7f947c0b1730 2024-03-25T19:24:03.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.983+0000 7f9472ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:03.988 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.987+0000 7f9472ffd640 10 client.?.objecter ms_handle_connect 0x7f947c0b4d60 2024-03-25T19:24:03.988 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.987+0000 7f9472ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:03.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.989+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749043561 0 0) 0x7f947c1362c0 con 0x7f947c0b1730 2024-03-25T19:24:03.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.989+0000 7f9472ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:03.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.989+0000 7f9472ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:03.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.989+0000 7f9472ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:03.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.989+0000 7f9472ffd640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:24:03.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.989+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9460004eb0 con 0x7f947c0b1730 2024-03-25T19:24:03.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.990+0000 7f9472ffd640 10 client.?.objecter ms_handle_connect 0x7f947c14be90 2024-03-25T19:24:03.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.990+0000 7f9472ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:03.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.998+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 585388167 0 0) 0x7f9460004eb0 con 0x7f947c0b1730 2024-03-25T19:24:03.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.998+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f947c1362c0 con 0x7f947c0b1730 2024-03-25T19:24:03.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.998+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9478001c30 con 0x7f947c0b1730 2024-03-25T19:24:03.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.998+0000 7f9472ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:03.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.998+0000 7f9472ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:03.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.998+0000 7f9472ffd640 10 monclient(hunting): dump: 2024-03-25T19:24:03.999 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:03.999 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:03.999 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:03.999 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:04.000 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:04.000 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:04.000 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:04.000 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:04.000 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:04.000 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:04.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:03.998+0000 7f9472ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:04.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1182426794 0 0) 0x7f947c1362c0 con 0x7f947c0b1730 2024-03-25T19:24:04.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 >> v1:172.21.15.123:6790/0 conn(0x7f947c0b4d60 legacy=0x7f947c1db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 >> v1:172.21.15.123:6789/0 conn(0x7f947c14be90 legacy=0x7f947c12fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:04.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 1 monclient: found mon.b 2024-03-25T19:24:04.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:04.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:24:04.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c143900 con 0x7f947c0b1730 2024-03-25T19:24:04.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 10 monclient: _finish_auth 0 2024-03-25T19:24:04.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:04.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f9472ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:04.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f948468d640 5 monclient: authenticate success, global_id 4300 2024-03-25T19:24:04.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f948468d640 10 monclient: _renew_subs 2024-03-25T19:24:04.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f948468d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:24:04.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f948468d640 1 -- 172.21.15.123:0/491891011 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947c1db710 con 0x7f947c0b1730 2024-03-25T19:24:04.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f948468d640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:04.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f948468d640 10 monclient: _renew_subs 2024-03-25T19:24:04.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f948468d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:24:04.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f948468d640 1 -- 172.21.15.123:0/491891011 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f947c130670 con 0x7f947c0b1730 2024-03-25T19:24:04.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.004+0000 7f948468d640 1 librados: init done 2024-03-25T19:24:04.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.005+0000 7f948468d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:04.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.005+0000 7f948468d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:04.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.005+0000 7f948468d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:24:04.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.005+0000 7f948468d640 1 -- 172.21.15.123:0/491891011 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947c0627b0 con 0x7f947c0b1730 2024-03-25T19:24:04.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.036+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9478004b60 con 0x7f947c0b1730 2024-03-25T19:24:04.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.036+0000 7f9472ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.036+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9478004fb0 con 0x7f947c0b1730 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.036+0000 7f9472ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.036+0000 7f9472ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.036+0000 7f9472ffd640 10 monclient: dump: 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:04.038 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:04.039 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:04.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.036+0000 7f9480c26640 4 set_mon_vals no callback set 2024-03-25T19:24:04.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.038+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f94780051d0 con 0x7f947c0b1730 2024-03-25T19:24:04.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.038+0000 7f9481c28640 10 monclient: get_auth_request con 0x7f946006dd00 auth_method 0 2024-03-25T19:24:04.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.038+0000 7f9472ffd640 10 client.4300.objecter ms_handle_connect 0x7f946006dd00 2024-03-25T19:24:04.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.041+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f9478054ba0 con 0x7f947c0b1730 2024-03-25T19:24:04.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.041+0000 7f9472ffd640 10 client.4300.objecter ms_dispatch 0x7f947c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:04.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.041+0000 7f9472ffd640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:04.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.041+0000 7f9472ffd640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:04.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.041+0000 7f9472ffd640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T19:24:04.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.303+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94780506b0 con 0x7f947c0b1730 2024-03-25T19:24:04.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.303+0000 7f9472ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:04.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.303+0000 7f9472ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:04.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.558+0000 7f948468d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:04.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.558+0000 7f948468d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:04.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.558+0000 7f948468d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:24:04.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.558+0000 7f948468d640 1 -- 172.21.15.123:0/491891011 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f947c1db8a0 con 0x7f947c0b1730 2024-03-25T19:24:04.584 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.583+0000 7f9472ffd640 1 -- 172.21.15.123:0/491891011 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f94780525e0 con 0x7f947c0b1730 2024-03-25T19:24:04.584 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.583+0000 7f9472ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:04.584 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.583+0000 7f9472ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:04.584 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:24:04.584 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.585+0000 7f948468d640 10 librados: watch_flush enter 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.585+0000 7f948468d640 10 librados: watch_flush exit 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.585+0000 7f948468d640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.585+0000 7f948468d640 10 client.4300.objecter successfully canceled tick 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.585+0000 7f948468d640 1 -- 172.21.15.123:0/491891011 >> v1:172.21.15.151:6800/727684239 conn(0x7f946006dd00 legacy=0x7f94600701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.585+0000 7f948468d640 10 monclient: shutdown 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.585+0000 7f948468d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.585+0000 7f948468d640 1 -- 172.21.15.123:0/491891011 >> v1:172.21.15.151:6789/0 conn(0x7f947c0b1730 legacy=0x7f947c136900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.585+0000 7f948468d640 1 -- 172.21.15.123:0/491891011 shutdown_connections 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.585+0000 7f948468d640 1 -- 172.21.15.123:0/491891011 >> 172.21.15.123:0/491891011 conn(0x7f947c0617a0 msgr2=0x7f947c13bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.586+0000 7f948468d640 1 -- 172.21.15.123:0/491891011 shutdown_connections 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.586+0000 7f948468d640 1 -- 172.21.15.123:0/491891011 wait complete. 2024-03-25T19:24:04.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:04.586+0000 7f948468d640 1 librados: shutdown 2024-03-25T19:24:04.598 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:09.751 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:01:59.323 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.799 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:01:59.371 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.824 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:01:59.396 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.852 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:01:59.385 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.856 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:01:59.428 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.856 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:01:59.389 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.867 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:01:59.439 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.890 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:01:59.423 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:09.891 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:01:59.463 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:11.405 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:11.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.523+0000 7f527a102640 10 monclient: get_monmap_and_config 2024-03-25T19:24:11.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.523+0000 7f527a102640 10 monclient: build_initial_monmap 2024-03-25T19:24:11.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.523+0000 7f527a102640 10 monclient: monmap: 2024-03-25T19:24:11.523 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:11.523 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:11.524 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:24:11.524556+0000 2024-03-25T19:24:11.524 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:24:11.524556+0000 2024-03-25T19:24:11.524 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:11.524 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:11.524 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:11.524 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:24:11.524 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:24:11.524 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:11.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.524+0000 7f527a102640 1 Processor -- start 2024-03-25T19:24:11.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.524+0000 7f527a102640 1 -- start start 2024-03-25T19:24:11.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.524+0000 7f527a102640 10 monclient: init 2024-03-25T19:24:11.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.525+0000 7f527a102640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:11.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.525+0000 7f527a102640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:11.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.525+0000 7f527a102640 10 monclient(hunting): picked mon.noname-c con 0x7f52740a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:11.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.525+0000 7f527a102640 10 monclient(hunting): picked mon.noname-a con 0x7f527413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:11.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.525+0000 7f527a102640 10 monclient(hunting): picked mon.noname-b con 0x7f52741430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:11.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.525+0000 7f527a102640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5274142520 con 0x7f527413f710 2024-03-25T19:24:11.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.525+0000 7f527a102640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52740b28a0 con 0x7f52740a98f0 2024-03-25T19:24:11.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.525+0000 7f527a102640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52740b2a30 con 0x7f52741430c0 2024-03-25T19:24:11.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.525+0000 7f527a102640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:11.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.525+0000 7f527a102640 10 monclient(hunting): authenticate will time out at 5688.885272s 2024-03-25T19:24:11.527 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.526+0000 7f52737fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f52740a98f0 0x7f52740a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:48636/0 (socket says 172.21.15.123:48636) 2024-03-25T19:24:11.527 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.526+0000 7f52737fe640 1 -- 172.21.15.123:0/3124731300 learned_addr learned my addr 172.21.15.123:0/3124731300 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:11.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.534+0000 7f52727fc640 1 -- 172.21.15.123:0/3124731300 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851633737 0 0) 0x7f52740b28a0 con 0x7f52740a98f0 2024-03-25T19:24:11.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.534+0000 7f52727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:11.534 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.534+0000 7f52727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:11.534 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.534+0000 7f52727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:11.534 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.534+0000 7f52727fc640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:24:11.534 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.534+0000 7f52727fc640 1 -- 172.21.15.123:0/3124731300 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525c003520 con 0x7f52740a98f0 2024-03-25T19:24:11.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 1 -- 172.21.15.123:0/3124731300 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2960915654 0 0) 0x7f525c003520 con 0x7f52740a98f0 2024-03-25T19:24:11.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 1 -- 172.21.15.123:0/3124731300 >> v1:172.21.15.123:6790/0 conn(0x7f52741430c0 legacy=0x7f527414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:11.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 1 -- 172.21.15.123:0/3124731300 >> v1:172.21.15.123:6789/0 conn(0x7f527413f710 legacy=0x7f52741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:11.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:11.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 1 monclient: found mon.noname-c 2024-03-25T19:24:11.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:11.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 1 -- 172.21.15.123:0/3124731300 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52740b44c0 con 0x7f52740a98f0 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 1 -- 172.21.15.123:0/3124731300 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f5264001690 con 0x7f52740a98f0 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f52727fc640 10 monclient: dump: 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:11.558 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:11.559 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:11.559 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:11.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f527a102640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:24:11.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.557+0000 7f527a102640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:11.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.565+0000 7f52727fc640 1 -- 172.21.15.123:0/3124731300 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52640017f0 con 0x7f52740a98f0 2024-03-25T19:24:11.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.565+0000 7f52727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:11.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.565+0000 7f52727fc640 1 -- 172.21.15.123:0/3124731300 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f5264001c00 con 0x7f52740a98f0 2024-03-25T19:24:11.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.565+0000 7f527a102640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:11.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.565+0000 7f527a102640 4 set_mon_vals no callback set 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.565+0000 7f52727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.565+0000 7f52727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.565+0000 7f52727fc640 10 monclient: dump: 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:11.566 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:11.567 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:11.567 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:11.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.566+0000 7f527a102640 10 monclient: shutdown 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.566+0000 7f527a102640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.566+0000 7f527a102640 1 -- 172.21.15.123:0/3124731300 >> v1:172.21.15.151:6789/0 conn(0x7f52740a98f0 legacy=0x7f52740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.566+0000 7f527a102640 1 -- 172.21.15.123:0/3124731300 shutdown_connections 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.566+0000 7f527a102640 1 -- 172.21.15.123:0/3124731300 >> 172.21.15.123:0/3124731300 conn(0x7f52740b7ea0 msgr2=0x7f52740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.566+0000 7f527a102640 1 -- 172.21.15.123:0/3124731300 shutdown_connections 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.566+0000 7f527a102640 1 -- 172.21.15.123:0/3124731300 wait complete. 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.566+0000 7f527a102640 10 monclient: build_initial_monmap 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.566+0000 7f527a102640 10 monclient: monmap: 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:11.569 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.567+0000 7f527a102640 1 librados: starting msgr at 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.567+0000 7f527a102640 1 librados: starting objecter 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.567+0000 7f527a102640 1 Processor -- start 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.567+0000 7f527a102640 1 -- start start 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.567+0000 7f527a102640 1 librados: setting wanted keys 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.567+0000 7f527a102640 1 librados: calling monclient init 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.567+0000 7f527a102640 10 monclient: init 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.568+0000 7f527a102640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.568+0000 7f527a102640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:24:11.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.568+0000 7f527a102640 10 monclient(hunting): picked mon.noname-c con 0x7f52740a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:11.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.568+0000 7f527a102640 10 monclient(hunting): picked mon.noname-b con 0x7f527413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:11.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.568+0000 7f527a102640 10 monclient(hunting): picked mon.noname-a con 0x7f52741430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:11.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.568+0000 7f527a102640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52740ab9e0 con 0x7f52741430c0 2024-03-25T19:24:11.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.568+0000 7f527a102640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5274138c20 con 0x7f527413f710 2024-03-25T19:24:11.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.568+0000 7f527a102640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5274138f00 con 0x7f52740a98f0 2024-03-25T19:24:11.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.568+0000 7f527a102640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:11.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.568+0000 7f527a102640 10 monclient(hunting): authenticate will time out at 5688.928199s 2024-03-25T19:24:11.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.570+0000 7f52737fe640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f52740a98f0 0x7f5274137fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:44406/0 (socket says 172.21.15.123:44406) 2024-03-25T19:24:11.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.570+0000 7f52737fe640 1 -- 172.21.15.123:0/2573102864 learned_addr learned my addr 172.21.15.123:0/2573102864 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:11.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.571+0000 7f525bfff640 10 client.?.objecter ms_handle_connect 0x7f52740a98f0 2024-03-25T19:24:11.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.571+0000 7f525bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:11.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.583+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193867593 0 0) 0x7f5274138f00 con 0x7f52740a98f0 2024-03-25T19:24:11.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.583+0000 7f525bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:11.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.583+0000 7f525bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:11.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.583+0000 7f525bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:11.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.583+0000 7f525bfff640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:24:11.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.583+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5260004eb0 con 0x7f52740a98f0 2024-03-25T19:24:11.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.588+0000 7f525bfff640 10 client.?.objecter ms_handle_connect 0x7f527413f710 2024-03-25T19:24:11.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.588+0000 7f525bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:11.592 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.592+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2212974532 0 0) 0x7f5260004eb0 con 0x7f52740a98f0 2024-03-25T19:24:11.592 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.592+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5274138f00 con 0x7f52740a98f0 2024-03-25T19:24:11.592 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.592+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f5264001bc0 con 0x7f52740a98f0 2024-03-25T19:24:11.592 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.592+0000 7f525bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:11.592 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.592+0000 7f525bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:11.592 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.592+0000 7f525bfff640 10 monclient(hunting): dump: 2024-03-25T19:24:11.592 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:11.593 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:11.593 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:11.593 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:11.593 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:11.593 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:11.593 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:11.593 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:11.593 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:11.593 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:11.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.592+0000 7f525bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:11.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.604+0000 7f525bfff640 10 client.?.objecter ms_handle_connect 0x7f52741430c0 2024-03-25T19:24:11.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.604+0000 7f525bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:11.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.604+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24401307 0 0) 0x7f5274138c20 con 0x7f527413f710 2024-03-25T19:24:11.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.604+0000 7f525bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:11.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.604+0000 7f525bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:11.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.604+0000 7f525bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:11.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.604+0000 7f525bfff640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:24:11.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.604+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5260004eb0 con 0x7f527413f710 2024-03-25T19:24:11.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.604+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1027637619 0 0) 0x7f5274138f00 con 0x7f52740a98f0 2024-03-25T19:24:11.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 >> v1:172.21.15.151:6789/0 conn(0x7f527413f710 legacy=0x7f52741d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:11.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 >> v1:172.21.15.123:6789/0 conn(0x7f52741430c0 legacy=0x7f52741e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:11.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f525bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:24:11.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f525bfff640 1 monclient: found mon.c 2024-03-25T19:24:11.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f525bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:11.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f525bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:24:11.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527414e7a0 con 0x7f52740a98f0 2024-03-25T19:24:11.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f525bfff640 10 monclient: _finish_auth 0 2024-03-25T19:24:11.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f525bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f525bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f527a102640 5 monclient: authenticate success, global_id 4313 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f527a102640 10 monclient: _renew_subs 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f527a102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f527a102640 1 -- 172.21.15.123:0/2573102864 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52741386d0 con 0x7f52740a98f0 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f527a102640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f527a102640 10 monclient: _renew_subs 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f527a102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f527a102640 1 -- 172.21.15.123:0/2573102864 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52741d8060 con 0x7f52740a98f0 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.605+0000 7f527a102640 1 librados: init done 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.606+0000 7f527a102640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.606+0000 7f527a102640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.606+0000 7f527a102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:24:11.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.606+0000 7f527a102640 1 -- 172.21.15.123:0/2573102864 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52740627b0 con 0x7f52740a98f0 2024-03-25T19:24:11.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.926+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5264004a90 con 0x7f52740a98f0 2024-03-25T19:24:11.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.926+0000 7f525bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:11.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.926+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f5264004ee0 con 0x7f52740a98f0 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.926+0000 7f525bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.926+0000 7f525bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.926+0000 7f525bfff640 10 monclient: dump: 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:11.927 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:11.928 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:11.928 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:11.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.926+0000 7f5271ffb640 4 set_mon_vals no callback set 2024-03-25T19:24:11.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.927+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f5264005100 con 0x7f52740a98f0 2024-03-25T19:24:11.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.927+0000 7f5272ffd640 10 monclient: get_auth_request con 0x7f526006dd90 auth_method 0 2024-03-25T19:24:11.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.928+0000 7f525bfff640 10 client.4313.objecter ms_handle_connect 0x7f526006dd90 2024-03-25T19:24:11.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.936+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f52640548f0 con 0x7f52740a98f0 2024-03-25T19:24:11.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.936+0000 7f525bfff640 10 client.4313.objecter ms_dispatch 0x7f5274000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:11.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.936+0000 7f525bfff640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:11.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.936+0000 7f525bfff640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:11.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.936+0000 7f525bfff640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T19:24:11.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.936+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5264050400 con 0x7f52740a98f0 2024-03-25T19:24:11.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.936+0000 7f525bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:11.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:11.936+0000 7f525bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:12.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.183+0000 7f527a102640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:12.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.183+0000 7f527a102640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:12.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.183+0000 7f527a102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:24:12.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.183+0000 7f527a102640 1 -- 172.21.15.123:0/2573102864 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f527414f240 con 0x7f52740a98f0 2024-03-25T19:24:12.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.218+0000 7f525bfff640 1 -- 172.21.15.123:0/2573102864 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f5264001210 con 0x7f52740a98f0 2024-03-25T19:24:12.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.218+0000 7f525bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:12.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.218+0000 7f525bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:12.218 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:24:12.218 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:12.220 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 10 librados: watch_flush enter 2024-03-25T19:24:12.220 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 10 librados: watch_flush exit 2024-03-25T19:24:12.220 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T19:24:12.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 10 client.4313.objecter successfully canceled tick 2024-03-25T19:24:12.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 1 -- 172.21.15.123:0/2573102864 >> v1:172.21.15.151:6800/727684239 conn(0x7f526006dd90 legacy=0x7f5260070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:12.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 10 monclient: shutdown 2024-03-25T19:24:12.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:12.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 1 -- 172.21.15.123:0/2573102864 >> v1:172.21.15.123:6790/0 conn(0x7f52740a98f0 legacy=0x7f5274137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:12.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 1 -- 172.21.15.123:0/2573102864 shutdown_connections 2024-03-25T19:24:12.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 1 -- 172.21.15.123:0/2573102864 >> 172.21.15.123:0/2573102864 conn(0x7f52740b7ea0 msgr2=0x7f5274074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:12.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 1 -- 172.21.15.123:0/2573102864 shutdown_connections 2024-03-25T19:24:12.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 1 -- 172.21.15.123:0/2573102864 wait complete. 2024-03-25T19:24:12.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:12.220+0000 7f527a102640 1 librados: shutdown 2024-03-25T19:24:12.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:19.041 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:19.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.156+0000 7f1613e2a640 10 monclient: get_monmap_and_config 2024-03-25T19:24:19.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.156+0000 7f1613e2a640 10 monclient: build_initial_monmap 2024-03-25T19:24:19.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.157+0000 7f1613e2a640 10 monclient: monmap: 2024-03-25T19:24:19.157 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:19.157 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:19.157 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:24:19.157915+0000 2024-03-25T19:24:19.157 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:24:19.157915+0000 2024-03-25T19:24:19.157 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:19.157 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:19.157 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:19.158 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:24:19.158 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:24:19.158 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:19.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.157+0000 7f1613e2a640 1 Processor -- start 2024-03-25T19:24:19.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 1 -- start start 2024-03-25T19:24:19.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 10 monclient: init 2024-03-25T19:24:19.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:19.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:19.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 10 monclient(hunting): picked mon.noname-a con 0x7f160c0a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:19.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 10 monclient(hunting): picked mon.noname-c con 0x7f160c13f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:19.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 10 monclient(hunting): picked mon.noname-b con 0x7f160c1430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:19.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c142520 con 0x7f160c0a98f0 2024-03-25T19:24:19.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c0b28a0 con 0x7f160c13f710 2024-03-25T19:24:19.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c0b2a30 con 0x7f160c1430c0 2024-03-25T19:24:19.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:19.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.158+0000 7f1613e2a640 10 monclient(hunting): authenticate will time out at 5696.518580s 2024-03-25T19:24:19.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.166+0000 7f1611bc6640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f160c0a98f0 0x7f160c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:58062/0 (socket says 172.21.15.123:58062) 2024-03-25T19:24:19.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.166+0000 7f1611bc6640 1 -- 172.21.15.123:0/1314938741 learned_addr learned my addr 172.21.15.123:0/1314938741 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:19.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.174+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065793139 0 0) 0x7f160c142520 con 0x7f160c0a98f0 2024-03-25T19:24:19.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.174+0000 7f1610bc4640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.174+0000 7f1610bc4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.174+0000 7f1610bc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.174+0000 7f1610bc4640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:24:19.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.174+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f8003520 con 0x7f160c0a98f0 2024-03-25T19:24:19.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.178+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749315737 0 0) 0x7f160c0b28a0 con 0x7f160c13f710 2024-03-25T19:24:19.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.178+0000 7f1610bc4640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.178+0000 7f1610bc4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.178+0000 7f1610bc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.178+0000 7f1610bc4640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:24:19.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.178+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160c142520 con 0x7f160c13f710 2024-03-25T19:24:19.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.179+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636688409 0 0) 0x7f160c0b2a30 con 0x7f160c1430c0 2024-03-25T19:24:19.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.179+0000 7f1610bc4640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.179+0000 7f1610bc4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.179+0000 7f1610bc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.179+0000 7f1610bc4640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:24:19.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.179+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160c0b28a0 con 0x7f160c1430c0 2024-03-25T19:24:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1413238580 0 0) 0x7f15f8003520 con 0x7f160c0a98f0 2024-03-25T19:24:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 >> v1:172.21.15.123:6790/0 conn(0x7f160c1430c0 legacy=0x7f160c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 >> v1:172.21.15.151:6789/0 conn(0x7f160c13f710 legacy=0x7f160c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 10 monclient: _finish_hunting 0 2024-03-25T19:24:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 1 monclient: found mon.noname-a 2024-03-25T19:24:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c0b44c0 con 0x7f160c0a98f0 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 10 monclient: _finish_auth 0 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 10 monclient: _check_auth_tickets 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1608001690 con 0x7f160c0a98f0 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1613e2a640 5 monclient: authenticate success, global_id 4290 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1613e2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1610bc4640 10 monclient: dump: 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:19.186 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:19.187 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.187 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.187 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.187 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:19.187 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:19.187 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:19.187 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:19.187 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:19.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.185+0000 7f1613e2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:19.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.193+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16080017f0 con 0x7f160c0a98f0 2024-03-25T19:24:19.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.193+0000 7f1610bc4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:19.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.193+0000 7f1610bc4640 1 -- 172.21.15.123:0/1314938741 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1608001c00 con 0x7f160c0a98f0 2024-03-25T19:24:19.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.193+0000 7f1613e2a640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:19.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.193+0000 7f1613e2a640 4 set_mon_vals no callback set 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.193+0000 7f1610bc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.193+0000 7f1610bc4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.193+0000 7f1610bc4640 10 monclient: dump: 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:19.194 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.194+0000 7f1613e2a640 10 monclient: shutdown 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.194+0000 7f1613e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.194+0000 7f1613e2a640 1 -- 172.21.15.123:0/1314938741 >> v1:172.21.15.123:6789/0 conn(0x7f160c0a98f0 legacy=0x7f160c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.194+0000 7f1613e2a640 1 -- 172.21.15.123:0/1314938741 shutdown_connections 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.194+0000 7f1613e2a640 1 -- 172.21.15.123:0/1314938741 >> 172.21.15.123:0/1314938741 conn(0x7f160c0b7ea0 msgr2=0x7f160c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.194+0000 7f1613e2a640 1 -- 172.21.15.123:0/1314938741 shutdown_connections 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.194+0000 7f1613e2a640 1 -- 172.21.15.123:0/1314938741 wait complete. 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.195+0000 7f1613e2a640 10 monclient: build_initial_monmap 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.195+0000 7f1613e2a640 10 monclient: monmap: 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:24:19.197 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.195+0000 7f1613e2a640 1 librados: starting msgr at 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.195+0000 7f1613e2a640 1 librados: starting objecter 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.195+0000 7f1613e2a640 1 Processor -- start 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.195+0000 7f1613e2a640 1 -- start start 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.195+0000 7f1613e2a640 1 librados: setting wanted keys 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.195+0000 7f1613e2a640 1 librados: calling monclient init 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.195+0000 7f1613e2a640 10 monclient: init 2024-03-25T19:24:19.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.196+0000 7f1613e2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.196+0000 7f1613e2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.196+0000 7f1613e2a640 10 monclient(hunting): picked mon.noname-b con 0x7f160c0a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.196+0000 7f1613e2a640 10 monclient(hunting): picked mon.noname-a con 0x7f160c13f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.196+0000 7f1613e2a640 10 monclient(hunting): picked mon.noname-c con 0x7f160c1430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.196+0000 7f1613e2a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c0abd30 con 0x7f160c13f710 2024-03-25T19:24:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.196+0000 7f1613e2a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c1d77a0 con 0x7f160c0a98f0 2024-03-25T19:24:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.196+0000 7f1613e2a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c1d51f0 con 0x7f160c1430c0 2024-03-25T19:24:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.196+0000 7f1613e2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.196+0000 7f1613e2a640 10 monclient(hunting): authenticate will time out at 5696.556598s 2024-03-25T19:24:19.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.202+0000 7f16113c5640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f160c13f710 0x7f160c138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:58096/0 (socket says 172.21.15.123:58096) 2024-03-25T19:24:19.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.202+0000 7f16113c5640 1 -- 172.21.15.123:0/3181629825 learned_addr learned my addr 172.21.15.123:0/3181629825 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:19.206 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.207+0000 7f16027fc640 10 client.?.objecter ms_handle_connect 0x7f160c1430c0 2024-03-25T19:24:19.206 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.207+0000 7f16027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:19.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.212+0000 7f16027fc640 10 client.?.objecter ms_handle_connect 0x7f160c13f710 2024-03-25T19:24:19.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.212+0000 7f16027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:19.219 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.219+0000 7f16027fc640 10 client.?.objecter ms_handle_connect 0x7f160c0a98f0 2024-03-25T19:24:19.219 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.219+0000 7f16027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:19.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.222+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814363175 0 0) 0x7f160c1d51f0 con 0x7f160c1430c0 2024-03-25T19:24:19.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.222+0000 7f16027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.222+0000 7f16027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.222+0000 7f16027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.222+0000 7f16027fc640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:24:19.223 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.222+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f0004eb0 con 0x7f160c1430c0 2024-03-25T19:24:19.224 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.224+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245895551 0 0) 0x7f160c0abd30 con 0x7f160c13f710 2024-03-25T19:24:19.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.224+0000 7f16027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.224+0000 7f16027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.224+0000 7f16027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.224+0000 7f16027fc640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:24:19.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.224+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160c1d51f0 con 0x7f160c13f710 2024-03-25T19:24:19.230 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.230+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723717139 0 0) 0x7f160c1d77a0 con 0x7f160c0a98f0 2024-03-25T19:24:19.230 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.230+0000 7f16027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.230 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.230+0000 7f16027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.230 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.230+0000 7f16027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.230 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.230+0000 7f16027fc640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:24:19.230 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.230+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160c0abd30 con 0x7f160c0a98f0 2024-03-25T19:24:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.239+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2746111177 0 0) 0x7f15f0004eb0 con 0x7f160c1430c0 2024-03-25T19:24:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.239+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f160c1d77a0 con 0x7f160c1430c0 2024-03-25T19:24:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.239+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1604001bd0 con 0x7f160c1430c0 2024-03-25T19:24:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.239+0000 7f16027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.239+0000 7f16027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.239+0000 7f16027fc640 10 monclient(hunting): dump: 2024-03-25T19:24:19.239 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:19.239 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:19.239 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.239 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.240 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.240 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:19.240 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:19.240 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:19.240 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:19.240 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:19.240 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.239+0000 7f16027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:19.245 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.244+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554674308 0 0) 0x7f160c1d51f0 con 0x7f160c13f710 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.244+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15f0004eb0 con 0x7f160c13f710 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.244+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f15fc0019b0 con 0x7f160c13f710 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.244+0000 7f16027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.244+0000 7f16027fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.244+0000 7f16027fc640 10 monclient(hunting): dump: 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:19.246 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:19.247 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:19.247 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:19.247 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:19.248 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.248+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4177107753 0 0) 0x7f160c0abd30 con 0x7f160c0a98f0 2024-03-25T19:24:19.248 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.248+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f160c1d51f0 con 0x7f160c0a98f0 2024-03-25T19:24:19.248 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.248+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1608001b00 con 0x7f160c0a98f0 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.248+0000 7f16027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.248+0000 7f16027fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.248+0000 7f16027fc640 10 monclient(hunting): dump: 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:19.249 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.250+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 305652465 0 0) 0x7f15f0004eb0 con 0x7f160c13f710 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 >> v1:172.21.15.123:6790/0 conn(0x7f160c1430c0 legacy=0x7f160c1e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 >> v1:172.21.15.151:6789/0 conn(0x7f160c0a98f0 legacy=0x7f160c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f16027fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f16027fc640 1 monclient: found mon.a 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f16027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f16027fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c1d83d0 con 0x7f160c13f710 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f16027fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f16027fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f16027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f1613e2a640 5 monclient: authenticate success, global_id 4293 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f1613e2a640 10 monclient: _renew_subs 2024-03-25T19:24:19.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f1613e2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:19.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f1613e2a640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160c138a50 con 0x7f160c13f710 2024-03-25T19:24:19.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f1613e2a640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:19.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f1613e2a640 10 monclient: _renew_subs 2024-03-25T19:24:19.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f1613e2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:19.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f1613e2a640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f160c138ee0 con 0x7f160c13f710 2024-03-25T19:24:19.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.251+0000 7f1613e2a640 1 librados: init done 2024-03-25T19:24:19.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.252+0000 7f1613e2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:19.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.252+0000 7f1613e2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:19.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.252+0000 7f1613e2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:19.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.252+0000 7f1613e2a640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160c074cc0 con 0x7f160c13f710 2024-03-25T19:24:19.543 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.543+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15fc0035f0 con 0x7f160c13f710 2024-03-25T19:24:19.543 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.543+0000 7f16027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.543+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f15fc003980 con 0x7f160c13f710 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.543+0000 7f16027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.543+0000 7f16027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.543+0000 7f16027fc640 10 monclient: dump: 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:19.544 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:19.545 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:19.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.543+0000 7f1603fff640 4 set_mon_vals no callback set 2024-03-25T19:24:19.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.544+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f15fc003ba0 con 0x7f160c13f710 2024-03-25T19:24:19.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.545+0000 7f1611bc6640 10 monclient: get_auth_request con 0x7f15f006e180 auth_method 0 2024-03-25T19:24:19.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.545+0000 7f16027fc640 10 client.4293.objecter ms_handle_connect 0x7f15f006e180 2024-03-25T19:24:19.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.553+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f15fc0547e0 con 0x7f160c13f710 2024-03-25T19:24:19.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.553+0000 7f16027fc640 10 client.4293.objecter ms_dispatch 0x7f160c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:19.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.553+0000 7f16027fc640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:19.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.553+0000 7f16027fc640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:19.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.553+0000 7f16027fc640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T19:24:19.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.553+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15fc0502f0 con 0x7f160c13f710 2024-03-25T19:24:19.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.554+0000 7f16027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:19.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.554+0000 7f16027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:19.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.802+0000 7f1613e2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:19.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.802+0000 7f1613e2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:19.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.802+0000 7f1613e2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:19.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.802+0000 7f1613e2a640 1 -- 172.21.15.123:0/3181629825 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f160c0627b0 con 0x7f160c13f710 2024-03-25T19:24:19.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.822+0000 7f16027fc640 1 -- 172.21.15.123:0/3181629825 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f15fc052220 con 0x7f160c13f710 2024-03-25T19:24:19.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.822+0000 7f16027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:19.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.822+0000 7f16027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:19.822 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:24:19.822 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:19.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 10 librados: watch_flush enter 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 10 librados: watch_flush exit 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 10 client.4293.objecter successfully canceled tick 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 1 -- 172.21.15.123:0/3181629825 >> v1:172.21.15.151:6800/727684239 conn(0x7f15f006e180 legacy=0x7f15f0070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 10 monclient: shutdown 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 1 -- 172.21.15.123:0/3181629825 >> v1:172.21.15.123:6789/0 conn(0x7f160c13f710 legacy=0x7f160c138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 1 -- 172.21.15.123:0/3181629825 shutdown_connections 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 1 -- 172.21.15.123:0/3181629825 >> 172.21.15.123:0/3181629825 conn(0x7f160c0b7ea0 msgr2=0x7f160c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 1 -- 172.21.15.123:0/3181629825 shutdown_connections 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 1 -- 172.21.15.123:0/3181629825 wait complete. 2024-03-25T19:24:19.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:19.824+0000 7f1613e2a640 1 librados: shutdown 2024-03-25T19:24:19.836 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:26.641 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.760+0000 7febab532640 10 monclient: get_monmap_and_config 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.760+0000 7febab532640 10 monclient: build_initial_monmap 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.760+0000 7febab532640 10 monclient: monmap: 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:24:26.761389+0000 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:24:26.761389+0000 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:26.760 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:24:26.761 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:24:26.761 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:26.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.761+0000 7febab532640 1 Processor -- start 2024-03-25T19:24:26.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.761+0000 7febab532640 1 -- start start 2024-03-25T19:24:26.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.761+0000 7febab532640 10 monclient: init 2024-03-25T19:24:26.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.762+0000 7febab532640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:26.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.762+0000 7febab532640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:26.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.762+0000 7febab532640 10 monclient(hunting): picked mon.noname-c con 0x7feba40a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:26.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.762+0000 7febab532640 10 monclient(hunting): picked mon.noname-a con 0x7feba413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:26.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.762+0000 7febab532640 10 monclient(hunting): picked mon.noname-b con 0x7feba41430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:26.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.762+0000 7febab532640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba4142520 con 0x7feba413f710 2024-03-25T19:24:26.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.762+0000 7febab532640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba40b28a0 con 0x7feba40a98f0 2024-03-25T19:24:26.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.762+0000 7febab532640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba40b2a30 con 0x7feba41430c0 2024-03-25T19:24:26.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.762+0000 7febab532640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:26.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.762+0000 7febab532640 10 monclient(hunting): authenticate will time out at 5704.122221s 2024-03-25T19:24:26.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.776+0000 7feba92ce640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7feba40a98f0 0x7feba40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:47804/0 (socket says 172.21.15.123:47804) 2024-03-25T19:24:26.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.776+0000 7feba92ce640 1 -- 172.21.15.123:0/1194917925 learned_addr learned my addr 172.21.15.123:0/1194917925 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:26.787 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.788+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184432845 0 0) 0x7feba40b2a30 con 0x7feba41430c0 2024-03-25T19:24:26.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.788+0000 7feb9bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:26.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.788+0000 7feb9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:26.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.788+0000 7feb9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:26.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.788+0000 7feb9bfff640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:24:26.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.788+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb90003520 con 0x7feba41430c0 2024-03-25T19:24:26.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.791+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741227444 0 0) 0x7feba40b28a0 con 0x7feba40a98f0 2024-03-25T19:24:26.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.791+0000 7feb9bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:26.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.791+0000 7feb9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:26.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.791+0000 7feb9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:26.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.791+0000 7feb9bfff640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:24:26.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.791+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba40b2a30 con 0x7feba40a98f0 2024-03-25T19:24:26.803 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.804+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380869016 0 0) 0x7feba4142520 con 0x7feba413f710 2024-03-25T19:24:26.803 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.804+0000 7feb9bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:26.804 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.804+0000 7feb9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:26.804 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.804+0000 7feb9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:26.804 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.804+0000 7feb9bfff640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:24:26.804 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.804+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba40b28a0 con 0x7feba413f710 2024-03-25T19:24:26.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1091881131 0 0) 0x7feba40b28a0 con 0x7feba413f710 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 >> v1:172.21.15.123:6790/0 conn(0x7feba41430c0 legacy=0x7feba414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 >> v1:172.21.15.151:6789/0 conn(0x7feba40a98f0 legacy=0x7feba40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 1 monclient: found mon.noname-a 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba40b44c0 con 0x7feba413f710 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 10 monclient: _finish_auth 0 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7feb94001570 con 0x7feba413f710 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7febab532640 5 monclient: authenticate success, global_id 4296 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7febab532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:26.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7feb9bfff640 10 monclient: dump: 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:26.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.879+0000 7febab532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:26.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.903+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb940016d0 con 0x7feba413f710 2024-03-25T19:24:26.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.903+0000 7feb9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:26.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.903+0000 7febab532640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:26.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.903+0000 7feb9bfff640 1 -- 172.21.15.123:0/1194917925 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7feb94001b20 con 0x7feba413f710 2024-03-25T19:24:26.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.903+0000 7febab532640 4 set_mon_vals no callback set 2024-03-25T19:24:26.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.903+0000 7feb9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:26.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.903+0000 7feb9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:26.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.903+0000 7feb9bfff640 10 monclient: dump: 2024-03-25T19:24:26.904 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:26.904 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:26.904 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:26.904 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:26.904 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:26.904 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:26.904 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:26.904 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:26.904 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:26.904 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:26.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.904+0000 7febab532640 10 monclient: shutdown 2024-03-25T19:24:26.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.904+0000 7febab532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:26.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.904+0000 7febab532640 1 -- 172.21.15.123:0/1194917925 >> v1:172.21.15.123:6789/0 conn(0x7feba413f710 legacy=0x7feba41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:26.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.904+0000 7febab532640 1 -- 172.21.15.123:0/1194917925 shutdown_connections 2024-03-25T19:24:26.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.904+0000 7febab532640 1 -- 172.21.15.123:0/1194917925 >> 172.21.15.123:0/1194917925 conn(0x7feba40b7ea0 msgr2=0x7feba40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:26.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.904+0000 7febab532640 1 -- 172.21.15.123:0/1194917925 shutdown_connections 2024-03-25T19:24:26.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.904+0000 7febab532640 1 -- 172.21.15.123:0/1194917925 wait complete. 2024-03-25T19:24:26.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.905+0000 7febab532640 10 monclient: build_initial_monmap 2024-03-25T19:24:26.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.905+0000 7febab532640 10 monclient: monmap: 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.905+0000 7febab532640 1 librados: starting msgr at 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.905+0000 7febab532640 1 librados: starting objecter 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.905+0000 7febab532640 1 Processor -- start 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.905+0000 7febab532640 1 -- start start 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.905+0000 7febab532640 1 librados: setting wanted keys 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.905+0000 7febab532640 1 librados: calling monclient init 2024-03-25T19:24:26.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.905+0000 7febab532640 10 monclient: init 2024-03-25T19:24:26.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.906+0000 7febab532640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:26.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.906+0000 7febab532640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:24:26.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.906+0000 7febab532640 10 monclient(hunting): picked mon.noname-c con 0x7feba40a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:26.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.906+0000 7febab532640 10 monclient(hunting): picked mon.noname-a con 0x7feba413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:26.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.906+0000 7febab532640 10 monclient(hunting): picked mon.noname-b con 0x7feba41430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:26.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.906+0000 7febab532640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba41d8340 con 0x7feba413f710 2024-03-25T19:24:26.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.906+0000 7febab532640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba41d7710 con 0x7feba41430c0 2024-03-25T19:24:26.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.906+0000 7febab532640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba4138e90 con 0x7feba40a98f0 2024-03-25T19:24:26.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.906+0000 7febab532640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:26.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.906+0000 7febab532640 10 monclient(hunting): authenticate will time out at 5704.266153s 2024-03-25T19:24:26.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.915+0000 7feba9acf640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7feba41430c0 0x7feba4133ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:47850/0 (socket says 172.21.15.123:47850) 2024-03-25T19:24:26.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.915+0000 7feba9acf640 1 -- 172.21.15.123:0/965142681 learned_addr learned my addr 172.21.15.123:0/965142681 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:26.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.915+0000 7feba8acd640 1 --1- 172.21.15.123:0/965142681 >> v1:172.21.15.123:6789/0 conn(0x7feba413f710 0x7feba4138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:51952/0 (socket says 172.21.15.123:51952) 2024-03-25T19:24:26.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.918+0000 7feb99ffb640 10 client.?.objecter ms_handle_connect 0x7feba413f710 2024-03-25T19:24:26.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.918+0000 7feb99ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:26.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.923+0000 7feb99ffb640 10 client.?.objecter ms_handle_connect 0x7feba41430c0 2024-03-25T19:24:26.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.923+0000 7feb99ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:26.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.934+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643454578 0 0) 0x7feba41d8340 con 0x7feba413f710 2024-03-25T19:24:26.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.934+0000 7feb99ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:24:26.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.934+0000 7feb99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:26.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.934+0000 7feb99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:26.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.934+0000 7feb99ffb640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:24:26.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.934+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb88004eb0 con 0x7feba413f710 2024-03-25T19:24:26.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.935+0000 7feb99ffb640 10 client.?.objecter ms_handle_connect 0x7feba40a98f0 2024-03-25T19:24:26.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.935+0000 7feb99ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:26.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.939+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1695547399 0 0) 0x7feb88004eb0 con 0x7feba413f710 2024-03-25T19:24:26.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.939+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feba41d8340 con 0x7feba413f710 2024-03-25T19:24:26.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.939+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7feb94001aa0 con 0x7feba413f710 2024-03-25T19:24:26.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.939+0000 7feb99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:26.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.939+0000 7feb99ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:26.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.939+0000 7feb99ffb640 10 monclient(hunting): dump: 2024-03-25T19:24:26.939 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:26.939 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:26.940 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:26.940 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:26.940 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:26.940 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:26.940 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:26.940 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:26.940 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:26.940 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:26.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.939+0000 7feb99ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2917683992 0 0) 0x7feba41d8340 con 0x7feba413f710 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 >> v1:172.21.15.123:6790/0 conn(0x7feba40a98f0 legacy=0x7feba41d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 >> v1:172.21.15.151:6789/0 conn(0x7feba41430c0 legacy=0x7feba4133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 1 monclient: found mon.a 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba4139020 con 0x7feba413f710 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 10 monclient: _finish_auth 0 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:24:26.961 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7feb99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:26.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7febab532640 5 monclient: authenticate success, global_id 4299 2024-03-25T19:24:26.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7febab532640 10 monclient: _renew_subs 2024-03-25T19:24:26.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7febab532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:26.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7febab532640 1 -- 172.21.15.123:0/965142681 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba4137f30 con 0x7feba413f710 2024-03-25T19:24:26.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7febab532640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:26.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7febab532640 10 monclient: _renew_subs 2024-03-25T19:24:26.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7febab532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:26.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7febab532640 1 -- 172.21.15.123:0/965142681 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feba41348e0 con 0x7feba413f710 2024-03-25T19:24:26.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.961+0000 7febab532640 1 librados: init done 2024-03-25T19:24:26.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.962+0000 7febab532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:26.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.962+0000 7febab532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:26.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.962+0000 7febab532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:26.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:26.962+0000 7febab532640 1 -- 172.21.15.123:0/965142681 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba41d78a0 con 0x7feba413f710 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.293+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb940038a0 con 0x7feba413f710 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.293+0000 7feb99ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.293+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7feb94003cf0 con 0x7feba413f710 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.293+0000 7feb99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.293+0000 7feb99ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.293+0000 7feb99ffb640 10 monclient: dump: 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:27.293 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:27.294 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:27.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.293+0000 7feb9b7fe640 4 set_mon_vals no callback set 2024-03-25T19:24:27.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.294+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7feb94003f10 con 0x7feba413f710 2024-03-25T19:24:27.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.295+0000 7feba92ce640 10 monclient: get_auth_request con 0x7feb8806dd00 auth_method 0 2024-03-25T19:24:27.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.295+0000 7feb99ffb640 10 client.4299.objecter ms_handle_connect 0x7feb8806dd00 2024-03-25T19:24:27.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.300+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7feb940548a0 con 0x7feba413f710 2024-03-25T19:24:27.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.300+0000 7feb99ffb640 10 client.4299.objecter ms_dispatch 0x7feba4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:27.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.300+0000 7feb99ffb640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:27.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.300+0000 7feb99ffb640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:27.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.300+0000 7feb99ffb640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-25T19:24:27.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.300+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb940503b0 con 0x7feba413f710 2024-03-25T19:24:27.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.300+0000 7feb99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:27.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.300+0000 7feb99ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:27.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.554+0000 7febab532640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:27.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.554+0000 7febab532640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:27.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.554+0000 7febab532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:27.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.554+0000 7febab532640 1 -- 172.21.15.123:0/965142681 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feba414f2d0 con 0x7feba413f710 2024-03-25T19:24:27.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.607+0000 7feb99ffb640 1 -- 172.21.15.123:0/965142681 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7feb940522e0 con 0x7feba413f710 2024-03-25T19:24:27.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.607+0000 7feb99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:27.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.607+0000 7feb99ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:27.607 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:24:27.607 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:27.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 10 librados: watch_flush enter 2024-03-25T19:24:27.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 10 librados: watch_flush exit 2024-03-25T19:24:27.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-25T19:24:27.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 10 client.4299.objecter successfully canceled tick 2024-03-25T19:24:27.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 1 -- 172.21.15.123:0/965142681 >> v1:172.21.15.151:6800/727684239 conn(0x7feb8806dd00 legacy=0x7feb880701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:27.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 10 monclient: shutdown 2024-03-25T19:24:27.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:27.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 1 -- 172.21.15.123:0/965142681 >> v1:172.21.15.123:6789/0 conn(0x7feba413f710 legacy=0x7feba4138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:27.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 1 -- 172.21.15.123:0/965142681 shutdown_connections 2024-03-25T19:24:27.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 1 -- 172.21.15.123:0/965142681 >> 172.21.15.123:0/965142681 conn(0x7feba40b7ea0 msgr2=0x7feba4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:27.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 1 -- 172.21.15.123:0/965142681 shutdown_connections 2024-03-25T19:24:27.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 1 -- 172.21.15.123:0/965142681 wait complete. 2024-03-25T19:24:27.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:27.610+0000 7febab532640 1 librados: shutdown 2024-03-25T19:24:27.622 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:29.785 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:02:19.357 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.833 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:02:19.405 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.868 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:02:19.441 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.901 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:02:19.435 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.905 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:02:19.477 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.907 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:02:19.440 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.915 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:02:19.487 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.937 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:02:19.471 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:29.943 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:02:19.516 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:34.427 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:34.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.541+0000 7f689941e640 10 monclient: get_monmap_and_config 2024-03-25T19:24:34.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.541+0000 7f689941e640 10 monclient: build_initial_monmap 2024-03-25T19:24:34.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.541+0000 7f689941e640 10 monclient: monmap: 2024-03-25T19:24:34.541 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:34.541 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:34.541 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:24:34.542527+0000 2024-03-25T19:24:34.542 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:24:34.542527+0000 2024-03-25T19:24:34.542 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:34.542 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:34.542 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:34.542 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:24:34.542 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:24:34.542 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:34.543 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.542+0000 7f689941e640 1 Processor -- start 2024-03-25T19:24:34.543 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.542+0000 7f689941e640 1 -- start start 2024-03-25T19:24:34.543 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.542+0000 7f689941e640 10 monclient: init 2024-03-25T19:24:34.543 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.543+0000 7f689941e640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:34.543 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.543+0000 7f689941e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:34.543 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.543+0000 7f689941e640 10 monclient(hunting): picked mon.noname-c con 0x7f68940a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:34.543 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.543+0000 7f689941e640 10 monclient(hunting): picked mon.noname-b con 0x7f689413f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:34.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.543+0000 7f689941e640 10 monclient(hunting): picked mon.noname-a con 0x7f68941430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:34.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.543+0000 7f689941e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6894142520 con 0x7f68941430c0 2024-03-25T19:24:34.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.543+0000 7f689941e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68940b28a0 con 0x7f68940a98f0 2024-03-25T19:24:34.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.543+0000 7f689941e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68940b2a30 con 0x7f689413f710 2024-03-25T19:24:34.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.543+0000 7f689941e640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:34.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.543+0000 7f689941e640 10 monclient(hunting): authenticate will time out at 5711.903030s 2024-03-25T19:24:34.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.554+0000 7f688bfff640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f689413f710 0x7f68941429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:54778/0 (socket says 172.21.15.123:54778) 2024-03-25T19:24:34.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.554+0000 7f688bfff640 1 -- 172.21.15.123:0/3423616987 learned_addr learned my addr 172.21.15.123:0/3423616987 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:34.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.564+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511099027 0 0) 0x7f68940b2a30 con 0x7f689413f710 2024-03-25T19:24:34.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.564+0000 7f68927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.564+0000 7f68927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.564+0000 7f68927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.564+0000 7f68927fc640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:24:34.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.564+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6880003520 con 0x7f689413f710 2024-03-25T19:24:34.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.569+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617504464 0 0) 0x7f68940b28a0 con 0x7f68940a98f0 2024-03-25T19:24:34.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.569+0000 7f68927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.569+0000 7f68927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.569+0000 7f68927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.569+0000 7f68927fc640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:24:34.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.569+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68940b2a30 con 0x7f68940a98f0 2024-03-25T19:24:34.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.576+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777190091 0 0) 0x7f6894142520 con 0x7f68941430c0 2024-03-25T19:24:34.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.576+0000 7f68927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.576+0000 7f68927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.576+0000 7f68927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.576+0000 7f68927fc640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:24:34.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.576+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68940b28a0 con 0x7f68941430c0 2024-03-25T19:24:34.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.579+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4034828148 0 0) 0x7f6880003520 con 0x7f689413f710 2024-03-25T19:24:34.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 >> v1:172.21.15.151:6789/0 conn(0x7f68940a98f0 legacy=0x7f68940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:34.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 >> v1:172.21.15.123:6789/0 conn(0x7f68941430c0 legacy=0x7f689414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:34.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:34.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 1 monclient: found mon.noname-b 2024-03-25T19:24:34.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:34.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:24:34.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68940b44c0 con 0x7f689413f710 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f687c001690 con 0x7f689413f710 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f689941e640 5 monclient: authenticate success, global_id 4325 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f689941e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f68927fc640 10 monclient: dump: 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:34.581 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:34.582 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:34.582 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:34.582 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:34.582 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:34.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.580+0000 7f689941e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.600+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f687c0017f0 con 0x7f689413f710 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.600+0000 7f68927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.600+0000 7f68927fc640 1 -- 172.21.15.123:0/3423616987 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f687c001c00 con 0x7f689413f710 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.600+0000 7f68927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.600+0000 7f68927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.600+0000 7f68927fc640 10 monclient: dump: 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.600 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:34.601 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:34.601 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:34.601 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:34.601 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:34.601 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:34.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.600+0000 7f689941e640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:34.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.600+0000 7f689941e640 4 set_mon_vals no callback set 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.601+0000 7f689941e640 10 monclient: shutdown 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.601+0000 7f689941e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.601+0000 7f689941e640 1 -- 172.21.15.123:0/3423616987 >> v1:172.21.15.123:6790/0 conn(0x7f689413f710 legacy=0x7f68941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.601+0000 7f689941e640 1 -- 172.21.15.123:0/3423616987 shutdown_connections 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.601+0000 7f689941e640 1 -- 172.21.15.123:0/3423616987 >> 172.21.15.123:0/3423616987 conn(0x7f68940b7ea0 msgr2=0x7f68940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.601+0000 7f689941e640 1 -- 172.21.15.123:0/3423616987 shutdown_connections 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.601+0000 7f689941e640 1 -- 172.21.15.123:0/3423616987 wait complete. 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 10 monclient: build_initial_monmap 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 10 monclient: monmap: 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:34.603 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 1 librados: starting msgr at 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 1 librados: starting objecter 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 1 Processor -- start 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 1 -- start start 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 1 librados: setting wanted keys 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 1 librados: calling monclient init 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 10 monclient: init 2024-03-25T19:24:34.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:34.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.602+0000 7f689941e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:34.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.603+0000 7f689941e640 10 monclient(hunting): picked mon.noname-b con 0x7f68940a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:34.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.603+0000 7f689941e640 10 monclient(hunting): picked mon.noname-c con 0x7f689413f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:34.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.603+0000 7f689941e640 10 monclient(hunting): picked mon.noname-a con 0x7f68941430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:34.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.603+0000 7f689941e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68940abc80 con 0x7f68941430c0 2024-03-25T19:24:34.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.603+0000 7f689941e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68941d7780 con 0x7f68940a98f0 2024-03-25T19:24:34.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.603+0000 7f689941e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68941399a0 con 0x7f689413f710 2024-03-25T19:24:34.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.603+0000 7f689941e640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:34.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.603+0000 7f689941e640 10 monclient(hunting): authenticate will time out at 5711.962932s 2024-03-25T19:24:34.609 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.609+0000 7f688bfff640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f689413f710 0x7f68941d4840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:54848/0 (socket says 172.21.15.123:54848) 2024-03-25T19:24:34.609 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.609+0000 7f688bfff640 1 -- 172.21.15.123:0/2674232684 learned_addr learned my addr 172.21.15.123:0/2674232684 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:34.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.610+0000 7f688b7fe640 10 client.?.objecter ms_handle_connect 0x7f689413f710 2024-03-25T19:24:34.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.610+0000 7f688b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:34.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.618+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163118036 0 0) 0x7f68941399a0 con 0x7f689413f710 2024-03-25T19:24:34.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.618+0000 7f688b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.618+0000 7f688b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.618+0000 7f688b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.618+0000 7f688b7fe640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:24:34.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.618+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6874004eb0 con 0x7f689413f710 2024-03-25T19:24:34.620 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.620+0000 7f688b7fe640 10 client.?.objecter ms_handle_connect 0x7f68940a98f0 2024-03-25T19:24:34.620 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.620+0000 7f688b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:34.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.623+0000 7f688b7fe640 10 client.?.objecter ms_handle_connect 0x7f68941430c0 2024-03-25T19:24:34.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.623+0000 7f688b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:34.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.631+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823394840 0 0) 0x7f68941d7780 con 0x7f68940a98f0 2024-03-25T19:24:34.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.631+0000 7f688b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.631+0000 7f688b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.631+0000 7f688b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.631+0000 7f688b7fe640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:24:34.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.631+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68941399a0 con 0x7f68940a98f0 2024-03-25T19:24:34.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.634+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502011290 0 0) 0x7f68940abc80 con 0x7f68941430c0 2024-03-25T19:24:34.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.634+0000 7f688b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.634+0000 7f688b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.635 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.634+0000 7f688b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.635 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.634+0000 7f688b7fe640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:24:34.635 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.634+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68941d7780 con 0x7f68941430c0 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.644+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1084082482 0 0) 0x7f6874004eb0 con 0x7f689413f710 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.644+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68940abc80 con 0x7f689413f710 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.644+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f687c001bc0 con 0x7f689413f710 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.644+0000 7f688b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.644+0000 7f688b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.644+0000 7f688b7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:34.644 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:34.645 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:34.645 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:34.645 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:34.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.644+0000 7f688b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:34.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.652+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2412800533 0 0) 0x7f68941d7780 con 0x7f68941430c0 2024-03-25T19:24:34.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.652+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6874004eb0 con 0x7f68941430c0 2024-03-25T19:24:34.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.652+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f688c001bd0 con 0x7f68941430c0 2024-03-25T19:24:34.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.652+0000 7f688b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:34.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.652+0000 7f688b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.652+0000 7f688b7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:34.653 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:34.660 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.660+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3954553262 0 0) 0x7f68941399a0 con 0x7f68940a98f0 2024-03-25T19:24:34.660 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.660+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68941d7780 con 0x7f68940a98f0 2024-03-25T19:24:34.660 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.660+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6884001970 con 0x7f68940a98f0 2024-03-25T19:24:34.660 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.660+0000 7f688b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:34.660 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.660+0000 7f688b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:34.660 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.660+0000 7f688b7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:34.660 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:34.661 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:34.661 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.661 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.661 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:34.661 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:34.661 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:34.661 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:34.661 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:34.661 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:34.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2411239321 0 0) 0x7f68940abc80 con 0x7f689413f710 2024-03-25T19:24:34.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 >> v1:172.21.15.151:6789/0 conn(0x7f68940a98f0 legacy=0x7f68941d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:34.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 >> v1:172.21.15.123:6789/0 conn(0x7f68941430c0 legacy=0x7f6894138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:34.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:24:34.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 1 monclient: found mon.c 2024-03-25T19:24:34.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:34.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:24:34.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68941d41e0 con 0x7f689413f710 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f688b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f689941e640 5 monclient: authenticate success, global_id 4328 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f689941e640 10 monclient: _renew_subs 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f689941e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f689941e640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68941d8370 con 0x7f689413f710 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f689941e640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f689941e640 10 monclient: _renew_subs 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f689941e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f689941e640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68941d5130 con 0x7f689413f710 2024-03-25T19:24:34.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.672+0000 7f689941e640 1 librados: init done 2024-03-25T19:24:34.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.673+0000 7f689941e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:34.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.673+0000 7f689941e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:34.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.673+0000 7f689941e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:24:34.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.673+0000 7f689941e640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68940627b0 con 0x7f689413f710 2024-03-25T19:24:34.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.978+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f687c0011f0 con 0x7f689413f710 2024-03-25T19:24:34.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.978+0000 7f688b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:34.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.978+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f687c004cb0 con 0x7f689413f710 2024-03-25T19:24:34.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.978+0000 7f688b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:34.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.978+0000 7f688b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:34.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.978+0000 7f688b7fe640 10 monclient: dump: 2024-03-25T19:24:34.978 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:34.978 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:34.979 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.979 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:34.979 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:34.979 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:34.979 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:34.979 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:34.979 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:34.979 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:34.979 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.978+0000 7f6891ffb640 4 set_mon_vals no callback set 2024-03-25T19:24:34.980 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.978+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f687c01ad00 con 0x7f689413f710 2024-03-25T19:24:34.980 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.979+0000 7f6892ffd640 10 monclient: get_auth_request con 0x7f687406e060 auth_method 0 2024-03-25T19:24:34.980 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.979+0000 7f688b7fe640 10 client.4328.objecter ms_handle_connect 0x7f687406e060 2024-03-25T19:24:34.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.984+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f687c0549e0 con 0x7f689413f710 2024-03-25T19:24:34.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.984+0000 7f688b7fe640 10 client.4328.objecter ms_dispatch 0x7f6894000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:34.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.984+0000 7f688b7fe640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:34.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.984+0000 7f688b7fe640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:34.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.984+0000 7f688b7fe640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-25T19:24:34.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.984+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f687c050320 con 0x7f689413f710 2024-03-25T19:24:34.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.984+0000 7f688b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:34.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:34.984+0000 7f688b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:35.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.195+0000 7f689941e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:35.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.195+0000 7f689941e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:35.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.195+0000 7f689941e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:24:35.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.195+0000 7f689941e640 1 -- 172.21.15.123:0/2674232684 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f689413e2b0 con 0x7f689413f710 2024-03-25T19:24:35.262 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.262+0000 7f688b7fe640 1 -- 172.21.15.123:0/2674232684 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f687c052420 con 0x7f689413f710 2024-03-25T19:24:35.262 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.262+0000 7f688b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:35.263 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.262+0000 7f688b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:35.263 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:24:35.263 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:35.265 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 10 librados: watch_flush enter 2024-03-25T19:24:35.265 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 10 librados: watch_flush exit 2024-03-25T19:24:35.265 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-25T19:24:35.265 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 10 client.4328.objecter successfully canceled tick 2024-03-25T19:24:35.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 1 -- 172.21.15.123:0/2674232684 >> v1:172.21.15.151:6800/727684239 conn(0x7f687406e060 legacy=0x7f6874070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:35.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 10 monclient: shutdown 2024-03-25T19:24:35.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:35.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 1 -- 172.21.15.123:0/2674232684 >> v1:172.21.15.123:6790/0 conn(0x7f689413f710 legacy=0x7f68941d4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:35.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 1 -- 172.21.15.123:0/2674232684 shutdown_connections 2024-03-25T19:24:35.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 1 -- 172.21.15.123:0/2674232684 >> 172.21.15.123:0/2674232684 conn(0x7f68940b7ea0 msgr2=0x7f6894074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:35.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 1 -- 172.21.15.123:0/2674232684 shutdown_connections 2024-03-25T19:24:35.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 1 -- 172.21.15.123:0/2674232684 wait complete. 2024-03-25T19:24:35.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:35.265+0000 7f689941e640 1 librados: shutdown 2024-03-25T19:24:35.276 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:42.082 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:42.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.198+0000 7fec9953b640 10 monclient: get_monmap_and_config 2024-03-25T19:24:42.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.198+0000 7fec9953b640 10 monclient: build_initial_monmap 2024-03-25T19:24:42.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.198+0000 7fec9953b640 10 monclient: monmap: 2024-03-25T19:24:42.199 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:42.199 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:42.200 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:24:42.200580+0000 2024-03-25T19:24:42.200 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:24:42.200580+0000 2024-03-25T19:24:42.200 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:42.200 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:42.200 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:42.200 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:24:42.200 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:24:42.200 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:42.201 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.199+0000 7fec9953b640 1 Processor -- start 2024-03-25T19:24:42.201 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.199+0000 7fec9953b640 1 -- start start 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.199+0000 7fec9953b640 10 monclient: init 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.200+0000 7fec9953b640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.200+0000 7fec9953b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.200+0000 7fec9953b640 10 monclient(hunting): picked mon.noname-a con 0x7fec940a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.200+0000 7fec9953b640 10 monclient(hunting): picked mon.noname-c con 0x7fec9413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.200+0000 7fec9953b640 10 monclient(hunting): picked mon.noname-b con 0x7fec941430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.200+0000 7fec9953b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec94142520 con 0x7fec940a98f0 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.200+0000 7fec9953b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec940b28a0 con 0x7fec9413f710 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.200+0000 7fec9953b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec940b2a30 con 0x7fec941430c0 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.200+0000 7fec9953b640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:42.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.200+0000 7fec9953b640 10 monclient(hunting): authenticate will time out at 5719.561598s 2024-03-25T19:24:42.207 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.206+0000 7fec937fe640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fec941430c0 0x7fec9414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:35268/0 (socket says 172.21.15.123:35268) 2024-03-25T19:24:42.207 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.207+0000 7fec937fe640 1 -- 172.21.15.123:0/2873580353 learned_addr learned my addr 172.21.15.123:0/2873580353 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:42.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.213+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527803530 0 0) 0x7fec940b2a30 con 0x7fec941430c0 2024-03-25T19:24:42.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.213+0000 7fec927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:42.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.213+0000 7fec927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:42.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.213+0000 7fec927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:42.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.213+0000 7fec927fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:24:42.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.213+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec80003520 con 0x7fec941430c0 2024-03-25T19:24:42.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.220+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357314062 0 0) 0x7fec940b28a0 con 0x7fec9413f710 2024-03-25T19:24:42.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.220+0000 7fec927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:42.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.220+0000 7fec927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:42.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.220+0000 7fec927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:42.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.220+0000 7fec927fc640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:24:42.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.220+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec940b2a30 con 0x7fec9413f710 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.229+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 40828668 0 0) 0x7fec80003520 con 0x7fec941430c0 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 >> v1:172.21.15.151:6789/0 conn(0x7fec9413f710 legacy=0x7fec941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 >> v1:172.21.15.123:6789/0 conn(0x7fec940a98f0 legacy=0x7fec940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 1 monclient: found mon.noname-b 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec940b44c0 con 0x7fec941430c0 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fec8c001a10 con 0x7fec941430c0 2024-03-25T19:24:42.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec9953b640 5 monclient: authenticate success, global_id 4331 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec9953b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec927fc640 10 monclient: dump: 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:42.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.230+0000 7fec9953b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.246+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec8c001b70 con 0x7fec941430c0 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.246+0000 7fec927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.246+0000 7fec927fc640 1 -- 172.21.15.123:0/2873580353 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fec8c001f80 con 0x7fec941430c0 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.246+0000 7fec9953b640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.246+0000 7fec9953b640 4 set_mon_vals no callback set 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.246+0000 7fec927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.246+0000 7fec927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.246+0000 7fec927fc640 10 monclient: dump: 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:42.247 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:42.248 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:42.248 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:42.248 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:42.248 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:42.248 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:42.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.247+0000 7fec9953b640 10 monclient: shutdown 2024-03-25T19:24:42.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.247+0000 7fec9953b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:42.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.247+0000 7fec9953b640 1 -- 172.21.15.123:0/2873580353 >> v1:172.21.15.123:6790/0 conn(0x7fec941430c0 legacy=0x7fec9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.247+0000 7fec9953b640 1 -- 172.21.15.123:0/2873580353 shutdown_connections 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.247+0000 7fec9953b640 1 -- 172.21.15.123:0/2873580353 >> 172.21.15.123:0/2873580353 conn(0x7fec940b7ea0 msgr2=0x7fec940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.247+0000 7fec9953b640 1 -- 172.21.15.123:0/2873580353 shutdown_connections 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.247+0000 7fec9953b640 1 -- 172.21.15.123:0/2873580353 wait complete. 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.248+0000 7fec9953b640 10 monclient: build_initial_monmap 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.248+0000 7fec9953b640 10 monclient: monmap: 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:42.250 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:24:42.251 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:24:42.251 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:42.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.248+0000 7fec9953b640 1 librados: starting msgr at 2024-03-25T19:24:42.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.248+0000 7fec9953b640 1 librados: starting objecter 2024-03-25T19:24:42.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.248+0000 7fec9953b640 1 Processor -- start 2024-03-25T19:24:42.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.248+0000 7fec9953b640 1 -- start start 2024-03-25T19:24:42.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.248+0000 7fec9953b640 1 librados: setting wanted keys 2024-03-25T19:24:42.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.248+0000 7fec9953b640 1 librados: calling monclient init 2024-03-25T19:24:42.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.248+0000 7fec9953b640 10 monclient: init 2024-03-25T19:24:42.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.249+0000 7fec9953b640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:42.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.249+0000 7fec9953b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:24:42.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.249+0000 7fec9953b640 10 monclient(hunting): picked mon.noname-a con 0x7fec940a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:42.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.249+0000 7fec9953b640 10 monclient(hunting): picked mon.noname-c con 0x7fec9413f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:42.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.249+0000 7fec9953b640 10 monclient(hunting): picked mon.noname-b con 0x7fec941430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:42.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.249+0000 7fec9953b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec941369a0 con 0x7fec940a98f0 2024-03-25T19:24:42.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.249+0000 7fec9953b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec941d8210 con 0x7fec941430c0 2024-03-25T19:24:42.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.249+0000 7fec9953b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec940abc30 con 0x7fec9413f710 2024-03-25T19:24:42.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.249+0000 7fec9953b640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:42.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.249+0000 7fec9953b640 10 monclient(hunting): authenticate will time out at 5719.610805s 2024-03-25T19:24:42.254 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.253+0000 7fec92ffd640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fec940a98f0 0x7fec941d7b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:36026/0 (socket says 172.21.15.123:36026) 2024-03-25T19:24:42.254 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.253+0000 7fec92ffd640 1 -- 172.21.15.123:0/365552775 learned_addr learned my addr 172.21.15.123:0/365552775 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:42.255 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.254+0000 7fec8b7fe640 10 client.?.objecter ms_handle_connect 0x7fec940a98f0 2024-03-25T19:24:42.255 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.254+0000 7fec8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:42.257 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.256+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974585369 0 0) 0x7fec94142520 con 0x7fec940a98f0 2024-03-25T19:24:42.257 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.256+0000 7fec8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:42.257 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.256+0000 7fec8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:42.257 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.256+0000 7fec8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:42.257 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.256+0000 7fec8b7fe640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:24:42.257 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.256+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec74004eb0 con 0x7fec940a98f0 2024-03-25T19:24:42.265 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.264+0000 7fec8b7fe640 10 client.?.objecter ms_handle_connect 0x7fec941430c0 2024-03-25T19:24:42.265 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.264+0000 7fec8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:42.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.268+0000 7fec8b7fe640 10 client.?.objecter ms_handle_connect 0x7fec9413f710 2024-03-25T19:24:42.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.268+0000 7fec8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:42.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.271+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1370770342 0 0) 0x7fec74004eb0 con 0x7fec940a98f0 2024-03-25T19:24:42.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.271+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec94142520 con 0x7fec940a98f0 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.271+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fec84001890 con 0x7fec940a98f0 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.271+0000 7fec8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.271+0000 7fec8b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.271+0000 7fec8b7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:42.273 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:42.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.271+0000 7fec8b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3455687647 0 0) 0x7fec94142520 con 0x7fec940a98f0 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 >> v1:172.21.15.123:6790/0 conn(0x7fec9413f710 legacy=0x7fec94138700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 >> v1:172.21.15.151:6789/0 conn(0x7fec941430c0 legacy=0x7fec941e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 1 monclient: found mon.a 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec94137ad0 con 0x7fec940a98f0 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:42.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec9953b640 5 monclient: authenticate success, global_id 4311 2024-03-25T19:24:42.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec9953b640 10 monclient: _renew_subs 2024-03-25T19:24:42.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec9953b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:42.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec9953b640 1 -- 172.21.15.123:0/365552775 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec94138e10 con 0x7fec940a98f0 2024-03-25T19:24:42.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec9953b640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:42.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec9953b640 10 monclient: _renew_subs 2024-03-25T19:24:42.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec9953b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:42.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec9953b640 1 -- 172.21.15.123:0/365552775 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec94137f90 con 0x7fec940a98f0 2024-03-25T19:24:42.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.280+0000 7fec9953b640 1 librados: init done 2024-03-25T19:24:42.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.281+0000 7fec9953b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:42.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.281+0000 7fec9953b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:42.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.281+0000 7fec9953b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:42.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.281+0000 7fec9953b640 1 -- 172.21.15.123:0/365552775 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec940627b0 con 0x7fec940a98f0 2024-03-25T19:24:42.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.592+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec840019f0 con 0x7fec940a98f0 2024-03-25T19:24:42.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.592+0000 7fec8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:42.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.592+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fec840037b0 con 0x7fec940a98f0 2024-03-25T19:24:42.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.592+0000 7fec8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:42.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.592+0000 7fec8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.592+0000 7fec8b7fe640 10 monclient: dump: 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:42.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.592+0000 7fec91ffb640 4 set_mon_vals no callback set 2024-03-25T19:24:42.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.593+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fec840039d0 con 0x7fec940a98f0 2024-03-25T19:24:42.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.594+0000 7fec8bfff640 10 monclient: get_auth_request con 0x7fec7406dc90 auth_method 0 2024-03-25T19:24:42.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.594+0000 7fec8b7fe640 10 client.4311.objecter ms_handle_connect 0x7fec7406dc90 2024-03-25T19:24:42.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.600+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fec84054770 con 0x7fec940a98f0 2024-03-25T19:24:42.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.601+0000 7fec8b7fe640 10 client.4311.objecter ms_dispatch 0x7fec94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:42.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.601+0000 7fec8b7fe640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:42.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.601+0000 7fec8b7fe640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:42.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.601+0000 7fec8b7fe640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-25T19:24:42.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.601+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec840500b0 con 0x7fec940a98f0 2024-03-25T19:24:42.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.601+0000 7fec8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:42.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.601+0000 7fec8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:42.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.857+0000 7fec9953b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:42.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.857+0000 7fec9953b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:42.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.857+0000 7fec9953b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:42.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.857+0000 7fec9953b640 1 -- 172.21.15.123:0/365552775 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec94069f40 con 0x7fec940a98f0 2024-03-25T19:24:42.887 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.886+0000 7fec8b7fe640 1 -- 172.21.15.123:0/365552775 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fec840521b0 con 0x7fec940a98f0 2024-03-25T19:24:42.887 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.886+0000 7fec8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:42.887 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.886+0000 7fec8b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:42.887 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:24:42.887 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:42.889 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 10 librados: watch_flush enter 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 10 librados: watch_flush exit 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 10 client.4311.objecter successfully canceled tick 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 1 -- 172.21.15.123:0/365552775 >> v1:172.21.15.151:6800/727684239 conn(0x7fec7406dc90 legacy=0x7fec74070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 10 monclient: shutdown 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 1 -- 172.21.15.123:0/365552775 >> v1:172.21.15.123:6789/0 conn(0x7fec940a98f0 legacy=0x7fec941d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 1 -- 172.21.15.123:0/365552775 shutdown_connections 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 1 -- 172.21.15.123:0/365552775 >> 172.21.15.123:0/365552775 conn(0x7fec940b7ea0 msgr2=0x7fec9413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 1 -- 172.21.15.123:0/365552775 shutdown_connections 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 1 -- 172.21.15.123:0/365552775 wait complete. 2024-03-25T19:24:42.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:42.888+0000 7fec9953b640 1 librados: shutdown 2024-03-25T19:24:42.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:49.706 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:49.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.810+0000 7f776f0cf640 10 monclient: get_monmap_and_config 2024-03-25T19:24:49.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.810+0000 7f776f0cf640 10 monclient: build_initial_monmap 2024-03-25T19:24:49.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.810+0000 7f776f0cf640 10 monclient: monmap: 2024-03-25T19:24:49.810 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:49.810 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:49.811 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:24:49.811626+0000 2024-03-25T19:24:49.811 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:24:49.811626+0000 2024-03-25T19:24:49.811 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:49.811 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:49.811 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:49.811 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:24:49.811 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:24:49.811 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:49.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.811+0000 7f776f0cf640 1 Processor -- start 2024-03-25T19:24:49.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.811+0000 7f776f0cf640 1 -- start start 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.811+0000 7f776f0cf640 10 monclient: init 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776f0cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776f0cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776f0cf640 10 monclient(hunting): picked mon.noname-b con 0x7f77680a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776f0cf640 10 monclient(hunting): picked mon.noname-a con 0x7f776813f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776f0cf640 10 monclient(hunting): picked mon.noname-c con 0x7f77681430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776f0cf640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7768142520 con 0x7f776813f710 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776f0cf640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680b28a0 con 0x7f77681430c0 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776f0cf640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680b2a30 con 0x7f77680a98f0 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776f0cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776f0cf640 10 monclient(hunting): authenticate will time out at 5727.172000s 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776ce6b640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f77680a98f0 0x7f77680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:58386/0 (socket says 172.21.15.123:58386) 2024-03-25T19:24:49.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.812+0000 7f776ce6b640 1 -- 172.21.15.123:0/248778129 learned_addr learned my addr 172.21.15.123:0/248778129 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:49.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.834+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258629655 0 0) 0x7f77680b28a0 con 0x7f77681430c0 2024-03-25T19:24:49.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.834+0000 7f77677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:49.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.834+0000 7f77677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:49.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.834+0000 7f77677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:49.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.834+0000 7f77677fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:24:49.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.834+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7758003520 con 0x7f77681430c0 2024-03-25T19:24:49.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.836+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375943911 0 0) 0x7f77680b2a30 con 0x7f77680a98f0 2024-03-25T19:24:49.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.836+0000 7f77677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:49.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.836+0000 7f77677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:49.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.836+0000 7f77677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:49.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.836+0000 7f77677fe640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:24:49.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.836+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77680b28a0 con 0x7f77680a98f0 2024-03-25T19:24:49.910 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:02:39.482 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.920 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:02:39.492 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.934+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288275878 0 0) 0x7f7768142520 con 0x7f776813f710 2024-03-25T19:24:49.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.934+0000 7f77677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:49.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.934+0000 7f77677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:49.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.934+0000 7f77677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:49.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.934+0000 7f77677fe640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:24:49.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.934+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77680b2a30 con 0x7f776813f710 2024-03-25T19:24:49.939 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:02:39.511 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1381863748 0 0) 0x7f77680b2a30 con 0x7f776813f710 2024-03-25T19:24:49.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 >> v1:172.21.15.123:6790/0 conn(0x7f77680a98f0 legacy=0x7f77680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:49.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 >> v1:172.21.15.151:6789/0 conn(0x7f77681430c0 legacy=0x7f776814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:49.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:24:49.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 1 monclient: found mon.noname-a 2024-03-25T19:24:49.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77680b44c0 con 0x7f776813f710 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 10 monclient: _finish_auth 0 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7754001570 con 0x7f776813f710 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f776f0cf640 5 monclient: authenticate success, global_id 4314 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f776f0cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f77677fe640 10 monclient: dump: 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:49.966 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:49.967 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:49.967 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:49.967 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:49.967 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:49.967 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:49.967 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:49.967 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:49.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:49.965+0000 7f776f0cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:49.977 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:02:39.550 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.980 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:02:39.513 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.986 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:02:39.520 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.989 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:02:39.561 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:50.012 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:02:39.545 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:50.020 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:02:39.592 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:50.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.032+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77540016d0 con 0x7f776813f710 2024-03-25T19:24:50.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.032+0000 7f77677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:50.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.032+0000 7f77677fe640 1 -- 172.21.15.123:0/248778129 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7754001b20 con 0x7f776813f710 2024-03-25T19:24:50.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.032+0000 7f776f0cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:50.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.032+0000 7f776f0cf640 4 set_mon_vals no callback set 2024-03-25T19:24:50.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.032+0000 7f77677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:50.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.032+0000 7f77677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:50.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.032+0000 7f77677fe640 10 monclient: dump: 2024-03-25T19:24:50.032 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:50.033 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:50.033 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:50.033 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:50.033 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:50.033 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:50.033 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:50.033 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:50.033 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:50.033 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:50.034 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.033+0000 7f776f0cf640 10 monclient: shutdown 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.033+0000 7f776f0cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.033+0000 7f776f0cf640 1 -- 172.21.15.123:0/248778129 >> v1:172.21.15.123:6789/0 conn(0x7f776813f710 legacy=0x7f77681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.033+0000 7f776f0cf640 1 -- 172.21.15.123:0/248778129 shutdown_connections 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.033+0000 7f776f0cf640 1 -- 172.21.15.123:0/248778129 >> 172.21.15.123:0/248778129 conn(0x7f77680b7ea0 msgr2=0x7f77680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.033+0000 7f776f0cf640 1 -- 172.21.15.123:0/248778129 shutdown_connections 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.033+0000 7f776f0cf640 1 -- 172.21.15.123:0/248778129 wait complete. 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.034+0000 7f776f0cf640 10 monclient: build_initial_monmap 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.034+0000 7f776f0cf640 10 monclient: monmap: 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:24:50.035 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.034+0000 7f776f0cf640 1 librados: starting msgr at 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.034+0000 7f776f0cf640 1 librados: starting objecter 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.034+0000 7f776f0cf640 1 Processor -- start 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.034+0000 7f776f0cf640 1 -- start start 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.034+0000 7f776f0cf640 1 librados: setting wanted keys 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.034+0000 7f776f0cf640 1 librados: calling monclient init 2024-03-25T19:24:50.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.034+0000 7f776f0cf640 10 monclient: init 2024-03-25T19:24:50.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.035+0000 7f776f0cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:50.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.035+0000 7f776f0cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:50.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.035+0000 7f776f0cf640 10 monclient(hunting): picked mon.noname-b con 0x7f77680a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:50.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.036+0000 7f776f0cf640 10 monclient(hunting): picked mon.noname-c con 0x7f776813f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:50.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.036+0000 7f776f0cf640 10 monclient(hunting): picked mon.noname-a con 0x7f77681430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:50.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.036+0000 7f776f0cf640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776813a570 con 0x7f77681430c0 2024-03-25T19:24:50.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.036+0000 7f776f0cf640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776813a0e0 con 0x7f77680a98f0 2024-03-25T19:24:50.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.036+0000 7f776f0cf640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680b7c60 con 0x7f776813f710 2024-03-25T19:24:50.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.036+0000 7f776f0cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:50.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.036+0000 7f776f0cf640 10 monclient(hunting): authenticate will time out at 5727.396010s 2024-03-25T19:24:50.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.059+0000 7f776d66c640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f77681430c0 0x7f77681d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:38790/0 (socket says 172.21.15.123:38790) 2024-03-25T19:24:50.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.059+0000 7f776d66c640 1 -- 172.21.15.123:0/3307330829 learned_addr learned my addr 172.21.15.123:0/3307330829 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:50.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.063+0000 7f77657fa640 10 client.?.objecter ms_handle_connect 0x7f77681430c0 2024-03-25T19:24:50.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.063+0000 7f77657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:50.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.075+0000 7f77657fa640 10 client.?.objecter ms_handle_connect 0x7f776813f710 2024-03-25T19:24:50.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.075+0000 7f77657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:50.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.081+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984391437 0 0) 0x7f776813a570 con 0x7f77681430c0 2024-03-25T19:24:50.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.081+0000 7f77657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:24:50.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.081+0000 7f77657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:50.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.081+0000 7f77657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:50.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.081+0000 7f77657fa640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:24:50.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.081+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f774c004eb0 con 0x7f77681430c0 2024-03-25T19:24:50.087 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.087+0000 7f77657fa640 10 client.?.objecter ms_handle_connect 0x7f77680a98f0 2024-03-25T19:24:50.087 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.087+0000 7f77657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:50.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.088+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417271156 0 0) 0x7f77680b7c60 con 0x7f776813f710 2024-03-25T19:24:50.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.088+0000 7f77657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:24:50.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.088+0000 7f77657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:50.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.088+0000 7f77657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:50.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.088+0000 7f77657fa640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:24:50.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.088+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f776813a570 con 0x7f776813f710 2024-03-25T19:24:50.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.108+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358827017 0 0) 0x7f774c004eb0 con 0x7f77681430c0 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.109+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77680b7c60 con 0x7f77681430c0 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.109+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7760001c30 con 0x7f77681430c0 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.109+0000 7f77657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.109+0000 7f77657fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.109+0000 7f77657fa640 10 monclient(hunting): dump: 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:50.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.109+0000 7f77657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:50.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.119+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71181421 0 0) 0x7f776813a0e0 con 0x7f77680a98f0 2024-03-25T19:24:50.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.119+0000 7f77657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:24:50.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.119+0000 7f77657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:50.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.119+0000 7f77657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:50.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.119+0000 7f77657fa640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:24:50.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.119+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f774c004eb0 con 0x7f77680a98f0 2024-03-25T19:24:50.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1790333339 0 0) 0x7f77680b7c60 con 0x7f77681430c0 2024-03-25T19:24:50.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 >> v1:172.21.15.123:6790/0 conn(0x7f776813f710 legacy=0x7f776812d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 >> v1:172.21.15.151:6789/0 conn(0x7f77680a98f0 legacy=0x7f77680b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 10 monclient: _finish_hunting 0 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 1 monclient: found mon.a 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7768139d20 con 0x7f77681430c0 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 10 monclient: _finish_auth 0 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 10 monclient: _check_auth_tickets 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f77657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f776f0cf640 5 monclient: authenticate success, global_id 4317 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f776f0cf640 10 monclient: _renew_subs 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f776f0cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f776f0cf640 1 -- 172.21.15.123:0/3307330829 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776812d9e0 con 0x7f77681430c0 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f776f0cf640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f776f0cf640 10 monclient: _renew_subs 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f776f0cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:50.129 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f776f0cf640 1 -- 172.21.15.123:0/3307330829 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f776812f650 con 0x7f77681430c0 2024-03-25T19:24:50.129 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.127+0000 7f776f0cf640 1 librados: init done 2024-03-25T19:24:50.129 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.128+0000 7f776f0cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:50.129 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.128+0000 7f776f0cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:50.129 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.128+0000 7f776f0cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:50.129 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.128+0000 7f776f0cf640 1 -- 172.21.15.123:0/3307330829 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77680b3ef0 con 0x7f77681430c0 2024-03-25T19:24:50.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.468+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7760003990 con 0x7f77681430c0 2024-03-25T19:24:50.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.468+0000 7f77657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:50.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.468+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7760002100 con 0x7f77681430c0 2024-03-25T19:24:50.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.468+0000 7f77657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:50.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.468+0000 7f77657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.468+0000 7f77657fa640 10 monclient: dump: 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:50.470 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:50.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.468+0000 7f7766ffd640 4 set_mon_vals no callback set 2024-03-25T19:24:50.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.469+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f7760003e60 con 0x7f77681430c0 2024-03-25T19:24:50.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.470+0000 7f776ce6b640 10 monclient: get_auth_request con 0x7f774c06de00 auth_method 0 2024-03-25T19:24:50.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.470+0000 7f77657fa640 10 client.4317.objecter ms_handle_connect 0x7f774c06de00 2024-03-25T19:24:50.475 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.475+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f7760054a60 con 0x7f77681430c0 2024-03-25T19:24:50.475 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.475+0000 7f77657fa640 10 client.4317.objecter ms_dispatch 0x7f7768000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:50.475 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.475+0000 7f77657fa640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:50.475 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.475+0000 7f77657fa640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:50.475 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.475+0000 7f77657fa640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T19:24:50.475 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.475+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7760050570 con 0x7f77681430c0 2024-03-25T19:24:50.475 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.475+0000 7f77657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:50.475 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.475+0000 7f77657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:50.722 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.722+0000 7f776f0cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:50.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.722+0000 7f776f0cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:50.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.722+0000 7f776f0cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:50.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.722+0000 7f776f0cf640 1 -- 172.21.15.123:0/3307330829 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f776813a290 con 0x7f77681430c0 2024-03-25T19:24:50.763 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.763+0000 7f77657fa640 1 -- 172.21.15.123:0/3307330829 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f776813a290 con 0x7f77681430c0 2024-03-25T19:24:50.763 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.763+0000 7f77657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:50.763 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.763+0000 7f77657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:50.764 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:24:50.764 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:50.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.765+0000 7f776f0cf640 10 librados: watch_flush enter 2024-03-25T19:24:50.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 10 librados: watch_flush exit 2024-03-25T19:24:50.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T19:24:50.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 10 client.4317.objecter successfully canceled tick 2024-03-25T19:24:50.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 1 -- 172.21.15.123:0/3307330829 >> v1:172.21.15.151:6800/727684239 conn(0x7f774c06de00 legacy=0x7f774c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 10 monclient: shutdown 2024-03-25T19:24:50.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:50.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 1 -- 172.21.15.123:0/3307330829 >> v1:172.21.15.123:6789/0 conn(0x7f77681430c0 legacy=0x7f77681d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 1 -- 172.21.15.123:0/3307330829 shutdown_connections 2024-03-25T19:24:50.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 1 -- 172.21.15.123:0/3307330829 >> 172.21.15.123:0/3307330829 conn(0x7f77680b7ea0 msgr2=0x7f7768074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:50.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 1 -- 172.21.15.123:0/3307330829 shutdown_connections 2024-03-25T19:24:50.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 1 -- 172.21.15.123:0/3307330829 wait complete. 2024-03-25T19:24:50.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:50.766+0000 7f776f0cf640 1 librados: shutdown 2024-03-25T19:24:50.778 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:57.584 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:57.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.702+0000 7f9d92cf6640 10 monclient: get_monmap_and_config 2024-03-25T19:24:57.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.702+0000 7f9d92cf6640 10 monclient: build_initial_monmap 2024-03-25T19:24:57.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.702+0000 7f9d92cf6640 10 monclient: monmap: 2024-03-25T19:24:57.703 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:57.703 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:57.703 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:24:57.703737+0000 2024-03-25T19:24:57.703 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:24:57.703737+0000 2024-03-25T19:24:57.703 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:57.703 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:57.703 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:57.704 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:24:57.704 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:24:57.704 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:57.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.703+0000 7f9d92cf6640 1 Processor -- start 2024-03-25T19:24:57.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.703+0000 7f9d92cf6640 1 -- start start 2024-03-25T19:24:57.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.703+0000 7f9d92cf6640 10 monclient: init 2024-03-25T19:24:57.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.704+0000 7f9d92cf6640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:57.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.704+0000 7f9d92cf6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:24:57.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.704+0000 7f9d92cf6640 10 monclient(hunting): picked mon.noname-a con 0x7f9d8c0a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:57.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.704+0000 7f9d92cf6640 10 monclient(hunting): picked mon.noname-b con 0x7f9d8c13f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:57.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.704+0000 7f9d92cf6640 10 monclient(hunting): picked mon.noname-c con 0x7f9d8c1430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:57.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.704+0000 7f9d92cf6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c142520 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.704+0000 7f9d92cf6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0b28a0 con 0x7f9d8c1430c0 2024-03-25T19:24:57.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.704+0000 7f9d92cf6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0b2a30 con 0x7f9d8c13f710 2024-03-25T19:24:57.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.704+0000 7f9d92cf6640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:57.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.704+0000 7f9d92cf6640 10 monclient(hunting): authenticate will time out at 5735.064267s 2024-03-25T19:24:57.720 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.721+0000 7f9d90a92640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f9d8c0a98f0 0x7f9d8c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:44500/0 (socket says 172.21.15.123:44500) 2024-03-25T19:24:57.720 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.721+0000 7f9d90a92640 1 -- 172.21.15.123:0/3477915296 learned_addr learned my addr 172.21.15.123:0/3477915296 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:57.739 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.739+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386006147 0 0) 0x7f9d8c0b28a0 con 0x7f9d8c1430c0 2024-03-25T19:24:57.739 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.739+0000 7f9d8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.739 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.739+0000 7f9d8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.739 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.739+0000 7f9d8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.739 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.739+0000 7f9d8b7fe640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:24:57.739 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.739+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d7c003520 con 0x7f9d8c1430c0 2024-03-25T19:24:57.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.740+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611274504 0 0) 0x7f9d8c0b2a30 con 0x7f9d8c13f710 2024-03-25T19:24:57.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.740+0000 7f9d8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.740+0000 7f9d8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.740+0000 7f9d8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.740+0000 7f9d8b7fe640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:24:57.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.740+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c0b28a0 con 0x7f9d8c13f710 2024-03-25T19:24:57.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.747+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700414291 0 0) 0x7f9d8c142520 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.747+0000 7f9d8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.747+0000 7f9d8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.747+0000 7f9d8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.747+0000 7f9d8b7fe640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:24:57.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.747+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c0b2a30 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2786091686 0 0) 0x7f9d8c0b28a0 con 0x7f9d8c13f710 2024-03-25T19:24:57.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 >> v1:172.21.15.151:6789/0 conn(0x7f9d8c1430c0 legacy=0x7f9d8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:57.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 >> v1:172.21.15.123:6789/0 conn(0x7f9d8c0a98f0 legacy=0x7f9d8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:57.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:24:57.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 1 monclient: found mon.noname-b 2024-03-25T19:24:57.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c0b44c0 con 0x7f9d8c13f710 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9d78001670 con 0x7f9d8c13f710 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d8b7fe640 10 monclient: dump: 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:57.755 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:57.756 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:57.756 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:57.756 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:57.756 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:57.756 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:57.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d92cf6640 5 monclient: authenticate success, global_id 4343 2024-03-25T19:24:57.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.754+0000 7f9d92cf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:57.763 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.764+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d780017d0 con 0x7f9d8c13f710 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.764+0000 7f9d8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.764+0000 7f9d8b7fe640 1 -- 172.21.15.123:0/3477915296 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9d78001be0 con 0x7f9d8c13f710 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.764+0000 7f9d92cf6640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.764+0000 7f9d92cf6640 4 set_mon_vals no callback set 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.764+0000 7f9d8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.764+0000 7f9d8b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.764+0000 7f9d8b7fe640 10 monclient: dump: 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:57.764 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:57.765 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:57.765 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:57.765 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:57.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.765+0000 7f9d92cf6640 10 monclient: shutdown 2024-03-25T19:24:57.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.765+0000 7f9d92cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:57.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.765+0000 7f9d92cf6640 1 -- 172.21.15.123:0/3477915296 >> v1:172.21.15.123:6790/0 conn(0x7f9d8c13f710 legacy=0x7f9d8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:57.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.765+0000 7f9d92cf6640 1 -- 172.21.15.123:0/3477915296 shutdown_connections 2024-03-25T19:24:57.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.765+0000 7f9d92cf6640 1 -- 172.21.15.123:0/3477915296 >> 172.21.15.123:0/3477915296 conn(0x7f9d8c0b7ea0 msgr2=0x7f9d8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.765+0000 7f9d92cf6640 1 -- 172.21.15.123:0/3477915296 shutdown_connections 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.765+0000 7f9d92cf6640 1 -- 172.21.15.123:0/3477915296 wait complete. 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.766+0000 7f9d92cf6640 10 monclient: build_initial_monmap 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.766+0000 7f9d92cf6640 10 monclient: monmap: 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:57.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.766+0000 7f9d92cf6640 1 librados: starting msgr at 2024-03-25T19:24:57.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.766+0000 7f9d92cf6640 1 librados: starting objecter 2024-03-25T19:24:57.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.766+0000 7f9d92cf6640 1 Processor -- start 2024-03-25T19:24:57.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.766+0000 7f9d92cf6640 1 -- start start 2024-03-25T19:24:57.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.766+0000 7f9d92cf6640 1 librados: setting wanted keys 2024-03-25T19:24:57.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.766+0000 7f9d92cf6640 1 librados: calling monclient init 2024-03-25T19:24:57.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.766+0000 7f9d92cf6640 10 monclient: init 2024-03-25T19:24:57.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.767+0000 7f9d92cf6640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:57.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.767+0000 7f9d92cf6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:24:57.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.767+0000 7f9d92cf6640 10 monclient(hunting): picked mon.noname-a con 0x7f9d8c0a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:24:57.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.767+0000 7f9d92cf6640 10 monclient(hunting): picked mon.noname-c con 0x7f9d8c13f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:24:57.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.768+0000 7f9d92cf6640 10 monclient(hunting): picked mon.noname-b con 0x7f9d8c1430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:24:57.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.768+0000 7f9d92cf6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c138700 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.768+0000 7f9d92cf6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c138b90 con 0x7f9d8c1430c0 2024-03-25T19:24:57.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.768+0000 7f9d92cf6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0b16d0 con 0x7f9d8c13f710 2024-03-25T19:24:57.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.768+0000 7f9d92cf6640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:57.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.768+0000 7f9d92cf6640 10 monclient(hunting): authenticate will time out at 5735.127919s 2024-03-25T19:24:57.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.770+0000 7f9d90a92640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f9d8c0a98f0 0x7f9d8c137820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:44554/0 (socket says 172.21.15.123:44554) 2024-03-25T19:24:57.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.770+0000 7f9d90a92640 1 -- 172.21.15.123:0/1527782589 learned_addr learned my addr 172.21.15.123:0/1527782589 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:24:57.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.777+0000 7f9d897fa640 10 client.?.objecter ms_handle_connect 0x7f9d8c0a98f0 2024-03-25T19:24:57.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.777+0000 7f9d897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:57.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.783+0000 7f9d897fa640 10 client.?.objecter ms_handle_connect 0x7f9d8c13f710 2024-03-25T19:24:57.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.783+0000 7f9d897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:57.783 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.783+0000 7f9d897fa640 10 client.?.objecter ms_handle_connect 0x7f9d8c1430c0 2024-03-25T19:24:57.783 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.783+0000 7f9d897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:57.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.794+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987032534 0 0) 0x7f9d8c138700 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.794+0000 7f9d897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.794+0000 7f9d897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.795 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.794+0000 7f9d897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.795 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.794+0000 7f9d897fa640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:24:57.795 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.794+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d70004eb0 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.797+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965962272 0 0) 0x7f9d8c0b16d0 con 0x7f9d8c13f710 2024-03-25T19:24:57.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.797+0000 7f9d897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.797+0000 7f9d897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.797+0000 7f9d897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.797+0000 7f9d897fa640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:24:57.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.797+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c138700 con 0x7f9d8c13f710 2024-03-25T19:24:57.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.801+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184080536 0 0) 0x7f9d8c138b90 con 0x7f9d8c1430c0 2024-03-25T19:24:57.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.801+0000 7f9d897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.801+0000 7f9d897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.801+0000 7f9d897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.801+0000 7f9d897fa640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:24:57.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.801+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c0b16d0 con 0x7f9d8c1430c0 2024-03-25T19:24:57.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.805+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2345679921 0 0) 0x7f9d8c138700 con 0x7f9d8c13f710 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.806+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d8c138b90 con 0x7f9d8c13f710 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.806+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9d78001ae0 con 0x7f9d8c13f710 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.806+0000 7f9d897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.806+0000 7f9d897fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.806+0000 7f9d897fa640 10 monclient(hunting): dump: 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:57.806 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:57.807 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:57.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.806+0000 7f9d897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.809+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 173676193 0 0) 0x7f9d70004eb0 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.809+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d8c138700 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.809+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9d80001910 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.809+0000 7f9d897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.809+0000 7f9d897fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.809+0000 7f9d897fa640 10 monclient(hunting): dump: 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:57.809 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:57.810 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:57.810 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:57.810 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:57.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3977390522 0 0) 0x7f9d8c138700 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 >> v1:172.21.15.123:6790/0 conn(0x7f9d8c13f710 legacy=0x7f9d8c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 >> v1:172.21.15.151:6789/0 conn(0x7f9d8c1430c0 legacy=0x7f9d8c1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 10 monclient: _finish_hunting 0 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 1 monclient: found mon.a 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c138050 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 10 monclient: _finish_auth 0 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 10 monclient: _check_auth_tickets 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d92cf6640 5 monclient: authenticate success, global_id 4323 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d92cf6640 10 monclient: _renew_subs 2024-03-25T19:24:57.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d92cf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:57.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d92cf6640 1 -- 172.21.15.123:0/1527782589 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c1382a0 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d92cf6640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:57.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d92cf6640 10 monclient: _renew_subs 2024-03-25T19:24:57.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d92cf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:57.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d92cf6640 1 -- 172.21.15.123:0/1527782589 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c1db170 con 0x7f9d8c0a98f0 2024-03-25T19:24:57.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.820+0000 7f9d92cf6640 1 librados: init done 2024-03-25T19:24:57.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.821+0000 7f9d92cf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:57.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.821+0000 7f9d92cf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:57.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.821+0000 7f9d92cf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:57.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:57.821+0000 7f9d92cf6640 1 -- 172.21.15.123:0/1527782589 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c1db420 con 0x7f9d8c0a98f0 2024-03-25T19:24:58.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.121+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d800047e0 con 0x7f9d8c0a98f0 2024-03-25T19:24:58.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.121+0000 7f9d897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:58.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.121+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9d80004c30 con 0x7f9d8c0a98f0 2024-03-25T19:24:58.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.121+0000 7f9d897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:58.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.121+0000 7f9d897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:58.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.121+0000 7f9d897fa640 10 monclient: dump: 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:24:58.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.121+0000 7f9d8affd640 4 set_mon_vals no callback set 2024-03-25T19:24:58.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.122+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f9d80004e50 con 0x7f9d8c0a98f0 2024-03-25T19:24:58.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.122+0000 7f9d8bfff640 10 monclient: get_auth_request con 0x7f9d7006df10 auth_method 0 2024-03-25T19:24:58.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.123+0000 7f9d897fa640 10 client.4323.objecter ms_handle_connect 0x7f9d7006df10 2024-03-25T19:24:58.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.142+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f9d80054880 con 0x7f9d8c0a98f0 2024-03-25T19:24:58.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.142+0000 7f9d897fa640 10 client.4323.objecter ms_dispatch 0x7f9d8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:58.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.142+0000 7f9d897fa640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:58.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.142+0000 7f9d897fa640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:58.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.142+0000 7f9d897fa640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T19:24:58.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.142+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d80050390 con 0x7f9d8c0a98f0 2024-03-25T19:24:58.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.142+0000 7f9d897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:58.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.142+0000 7f9d897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:58.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.394+0000 7f9d92cf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:58.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.394+0000 7f9d92cf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:58.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.394+0000 7f9d92cf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:24:58.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.394+0000 7f9d92cf6640 1 -- 172.21.15.123:0/1527782589 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d8c1384e0 con 0x7f9d8c0a98f0 2024-03-25T19:24:58.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.438+0000 7f9d897fa640 1 -- 172.21.15.123:0/1527782589 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f9d800522c0 con 0x7f9d8c0a98f0 2024-03-25T19:24:58.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.438+0000 7f9d897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:58.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.438+0000 7f9d897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:58.438 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:24:58.438 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:58.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.440+0000 7f9d92cf6640 10 librados: watch_flush enter 2024-03-25T19:24:58.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 10 librados: watch_flush exit 2024-03-25T19:24:58.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T19:24:58.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 10 client.4323.objecter successfully canceled tick 2024-03-25T19:24:58.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 1 -- 172.21.15.123:0/1527782589 >> v1:172.21.15.151:6800/727684239 conn(0x7f9d7006df10 legacy=0x7f9d700703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:58.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 10 monclient: shutdown 2024-03-25T19:24:58.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:58.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 1 -- 172.21.15.123:0/1527782589 >> v1:172.21.15.123:6789/0 conn(0x7f9d8c0a98f0 legacy=0x7f9d8c137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:58.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 1 -- 172.21.15.123:0/1527782589 shutdown_connections 2024-03-25T19:24:58.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 1 -- 172.21.15.123:0/1527782589 >> 172.21.15.123:0/1527782589 conn(0x7f9d8c0b7ea0 msgr2=0x7f9d8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:58.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 1 -- 172.21.15.123:0/1527782589 shutdown_connections 2024-03-25T19:24:58.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 1 -- 172.21.15.123:0/1527782589 wait complete. 2024-03-25T19:24:58.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:24:58.441+0000 7f9d92cf6640 1 librados: shutdown 2024-03-25T19:24:58.453 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:05.259 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:05.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.372+0000 7f01bd910640 10 monclient: get_monmap_and_config 2024-03-25T19:25:05.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.373+0000 7f01bd910640 10 monclient: build_initial_monmap 2024-03-25T19:25:05.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.373+0000 7f01bd910640 10 monclient: monmap: 2024-03-25T19:25:05.375 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:05.375 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:05.375 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:25:05.373937+0000 2024-03-25T19:25:05.375 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:25:05.373937+0000 2024-03-25T19:25:05.375 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:05.375 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:05.376 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:05.376 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:25:05.376 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:25:05.376 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:05.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.373+0000 7f01bd910640 1 Processor -- start 2024-03-25T19:25:05.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.373+0000 7f01bd910640 1 -- start start 2024-03-25T19:25:05.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.373+0000 7f01bd910640 10 monclient: init 2024-03-25T19:25:05.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.374+0000 7f01bd910640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:05.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.374+0000 7f01bd910640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:25:05.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.374+0000 7f01bd910640 10 monclient(hunting): picked mon.noname-c con 0x7f01b806c380 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:05.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.374+0000 7f01bd910640 10 monclient(hunting): picked mon.noname-a con 0x7f01b80b12d0 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:05.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.374+0000 7f01bd910640 10 monclient(hunting): picked mon.noname-b con 0x7f01b80aedf0 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:05.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.374+0000 7f01bd910640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b80b16f0 con 0x7f01b80b12d0 2024-03-25T19:25:05.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.374+0000 7f01bd910640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b80adfe0 con 0x7f01b806c380 2024-03-25T19:25:05.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.374+0000 7f01bd910640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b80add00 con 0x7f01b80aedf0 2024-03-25T19:25:05.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.374+0000 7f01bd910640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:05.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.374+0000 7f01bd910640 10 monclient(hunting): authenticate will time out at 5742.734506s 2024-03-25T19:25:05.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.381+0000 7f01b77fe640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f01b80aedf0 0x7f01b814e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:59410/0 (socket says 172.21.15.123:59410) 2024-03-25T19:25:05.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.381+0000 7f01b77fe640 1 -- 172.21.15.123:0/1966326207 learned_addr learned my addr 172.21.15.123:0/1966326207 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:05.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.389+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878646491 0 0) 0x7f01b80add00 con 0x7f01b80aedf0 2024-03-25T19:25:05.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.389+0000 7f01b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:05.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.389+0000 7f01b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:05.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.389+0000 7f01b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:05.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.389+0000 7f01b5ffb640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:25:05.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.389+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a0003520 con 0x7f01b80aedf0 2024-03-25T19:25:05.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.400+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233902009 0 0) 0x7f01b80adfe0 con 0x7f01b806c380 2024-03-25T19:25:05.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.400+0000 7f01b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:05.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.400+0000 7f01b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:05.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.400+0000 7f01b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:05.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.400+0000 7f01b5ffb640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:25:05.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.400+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b80add00 con 0x7f01b806c380 2024-03-25T19:25:05.411 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91873142 0 0) 0x7f01b80add00 con 0x7f01b806c380 2024-03-25T19:25:05.411 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 >> v1:172.21.15.123:6790/0 conn(0x7f01b80aedf0 legacy=0x7f01b814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.411 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 >> v1:172.21.15.123:6789/0 conn(0x7f01b80b12d0 legacy=0x7f01b80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.411 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:25:05.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:25:05.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:05.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:25:05.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b80ae2c0 con 0x7f01b806c380 2024-03-25T19:25:05.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:25:05.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:25:05.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:05.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f01a8001670 con 0x7f01b806c380 2024-03-25T19:25:05.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01bd910640 5 monclient: authenticate success, global_id 4342 2024-03-25T19:25:05.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01bd910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:05.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:05.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:05.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01b5ffb640 10 monclient: dump: 2024-03-25T19:25:05.413 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:05.413 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:05.413 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:05.414 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:05.414 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:05.414 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:05.414 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:05.414 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:05.414 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:05.414 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.410+0000 7f01bd910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:05.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.417+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01a80017d0 con 0x7f01b806c380 2024-03-25T19:25:05.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.417+0000 7f01b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:05.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.417+0000 7f01b5ffb640 1 -- 172.21.15.123:0/1966326207 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f01a8001b60 con 0x7f01b806c380 2024-03-25T19:25:05.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.417+0000 7f01b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:05.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.417+0000 7f01b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:05.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.417+0000 7f01b5ffb640 10 monclient: dump: 2024-03-25T19:25:05.419 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:05.419 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:05.419 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:05.419 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:05.419 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:05.419 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:05.419 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:05.419 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:05.420 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:05.420 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:05.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.417+0000 7f01bd910640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:05.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.417+0000 7f01bd910640 4 set_mon_vals no callback set 2024-03-25T19:25:05.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.418+0000 7f01bd910640 10 monclient: shutdown 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.418+0000 7f01bd910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.418+0000 7f01bd910640 1 -- 172.21.15.123:0/1966326207 >> v1:172.21.15.151:6789/0 conn(0x7f01b806c380 legacy=0x7f01b80b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.418+0000 7f01bd910640 1 -- 172.21.15.123:0/1966326207 shutdown_connections 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.418+0000 7f01bd910640 1 -- 172.21.15.123:0/1966326207 >> 172.21.15.123:0/1966326207 conn(0x7f01b80b5d10 msgr2=0x7f01b80b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.418+0000 7f01bd910640 1 -- 172.21.15.123:0/1966326207 shutdown_connections 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.419+0000 7f01bd910640 1 -- 172.21.15.123:0/1966326207 wait complete. 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.419+0000 7f01bd910640 10 monclient: build_initial_monmap 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.419+0000 7f01bd910640 10 monclient: monmap: 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:05.423 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.419+0000 7f01bd910640 1 librados: starting msgr at 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.420+0000 7f01bd910640 1 librados: starting objecter 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.420+0000 7f01bd910640 1 Processor -- start 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.420+0000 7f01bd910640 1 -- start start 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.420+0000 7f01bd910640 1 librados: setting wanted keys 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.420+0000 7f01bd910640 1 librados: calling monclient init 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.420+0000 7f01bd910640 10 monclient: init 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.420+0000 7f01bd910640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.420+0000 7f01bd910640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.420+0000 7f01bd910640 10 monclient(hunting): picked mon.noname-b con 0x7f01b806c380 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:05.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.420+0000 7f01bd910640 10 monclient(hunting): picked mon.noname-c con 0x7f01b80aedf0 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:05.425 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.421+0000 7f01bd910640 10 monclient(hunting): picked mon.noname-a con 0x7f01b80b12d0 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:05.425 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.421+0000 7f01bd910640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b806c8e0 con 0x7f01b80b12d0 2024-03-25T19:25:05.425 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.421+0000 7f01bd910640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b81c2f90 con 0x7f01b806c380 2024-03-25T19:25:05.425 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.421+0000 7f01bd910640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b81c1790 con 0x7f01b80aedf0 2024-03-25T19:25:05.425 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.421+0000 7f01bd910640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:05.425 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.421+0000 7f01bd910640 10 monclient(hunting): authenticate will time out at 5742.781000s 2024-03-25T19:25:05.425 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.424+0000 7f01b6ffd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f01b806c380 0x7f01b81c4890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:60212/0 (socket says 172.21.15.123:60212) 2024-03-25T19:25:05.425 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.424+0000 7f01b6ffd640 1 -- 172.21.15.123:0/3440908960 learned_addr learned my addr 172.21.15.123:0/3440908960 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:05.426 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.426+0000 7f019ffff640 10 client.?.objecter ms_handle_connect 0x7f01b80b12d0 2024-03-25T19:25:05.426 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.426+0000 7f019ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:05.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.427+0000 7f019ffff640 10 client.?.objecter ms_handle_connect 0x7f01b806c380 2024-03-25T19:25:05.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.427+0000 7f019ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:05.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.433+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234579025 0 0) 0x7f01b806c8e0 con 0x7f01b80b12d0 2024-03-25T19:25:05.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.433+0000 7f019ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:05.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.433+0000 7f019ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:05.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.433+0000 7f019ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:05.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.433+0000 7f019ffff640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:25:05.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.433+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a4004eb0 con 0x7f01b80b12d0 2024-03-25T19:25:05.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.437+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75095241 0 0) 0x7f01b81c2f90 con 0x7f01b806c380 2024-03-25T19:25:05.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.437+0000 7f019ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:05.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.437+0000 7f019ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:05.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.437+0000 7f019ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:05.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.437+0000 7f019ffff640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:25:05.437 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.437+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b806c8e0 con 0x7f01b806c380 2024-03-25T19:25:05.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.441+0000 7f019ffff640 10 client.?.objecter ms_handle_connect 0x7f01b80aedf0 2024-03-25T19:25:05.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.441+0000 7f019ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:05.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.446+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1410116125 0 0) 0x7f01a4004eb0 con 0x7f01b80b12d0 2024-03-25T19:25:05.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.446+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01b81c2f90 con 0x7f01b80b12d0 2024-03-25T19:25:05.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.446+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f01b0001cd0 con 0x7f01b80b12d0 2024-03-25T19:25:05.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.446+0000 7f019ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:05.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.446+0000 7f019ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:05.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.446+0000 7f019ffff640 10 monclient(hunting): dump: 2024-03-25T19:25:05.446 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:05.447 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:05.447 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:05.447 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:05.447 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:05.447 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:05.447 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:05.447 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:05.447 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:05.447 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:05.447 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.446+0000 7f019ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:05.449 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.449+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3728624789 0 0) 0x7f01b806c8e0 con 0x7f01b806c380 2024-03-25T19:25:05.449 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.449+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01a4004eb0 con 0x7f01b806c380 2024-03-25T19:25:05.449 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.449+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f01a8001ae0 con 0x7f01b806c380 2024-03-25T19:25:05.449 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.449+0000 7f019ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:05.449 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.449+0000 7f019ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:05.449 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.449+0000 7f019ffff640 10 monclient(hunting): dump: 2024-03-25T19:25:05.449 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:05.449 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:05.449 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:05.449 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:05.450 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:05.450 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:05.450 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:05.450 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:05.450 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:05.450 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:05.452 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.452+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095358950 0 0) 0x7f01b81c1790 con 0x7f01b80aedf0 2024-03-25T19:25:05.452 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.452+0000 7f019ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:05.452 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.452+0000 7f019ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:05.453 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.452+0000 7f019ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:05.453 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.452+0000 7f019ffff640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:25:05.453 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.452+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b806c8e0 con 0x7f01b80aedf0 2024-03-25T19:25:05.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 471101181 0 0) 0x7f01b81c2f90 con 0x7f01b80b12d0 2024-03-25T19:25:05.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 >> v1:172.21.15.123:6790/0 conn(0x7f01b80aedf0 legacy=0x7f01b81c03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 >> v1:172.21.15.151:6789/0 conn(0x7f01b806c380 legacy=0x7f01b81c4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:25:05.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 1 monclient: found mon.a 2024-03-25T19:25:05.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:05.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b81c5350 con 0x7f01b80b12d0 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 10 monclient: _finish_auth 0 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f019ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f01bd910640 5 monclient: authenticate success, global_id 4329 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f01bd910640 10 monclient: _renew_subs 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f01bd910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f01bd910640 1 -- 172.21.15.123:0/3440908960 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b81c55a0 con 0x7f01b80b12d0 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f01bd910640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f01bd910640 10 monclient: _renew_subs 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f01bd910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f01bd910640 1 -- 172.21.15.123:0/3440908960 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b81c0c90 con 0x7f01b80b12d0 2024-03-25T19:25:05.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.454+0000 7f01bd910640 1 librados: init done 2024-03-25T19:25:05.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.455+0000 7f01bd910640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:05.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.455+0000 7f01bd910640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:05.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.455+0000 7f01bd910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:25:05.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.455+0000 7f01bd910640 1 -- 172.21.15.123:0/3440908960 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b806d790 con 0x7f01b80b12d0 2024-03-25T19:25:05.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.476+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01b00020e0 con 0x7f01b80b12d0 2024-03-25T19:25:05.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.476+0000 7f019ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:05.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.476+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f01b0004b60 con 0x7f01b80b12d0 2024-03-25T19:25:05.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.476+0000 7f019ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:05.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.476+0000 7f019ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:05.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.476+0000 7f019ffff640 10 monclient: dump: 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:05.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.476+0000 7f01b57fa640 4 set_mon_vals no callback set 2024-03-25T19:25:05.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.477+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f01b001ab30 con 0x7f01b80b12d0 2024-03-25T19:25:05.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.478+0000 7f01b6ffd640 10 monclient: get_auth_request con 0x7f01a406df50 auth_method 0 2024-03-25T19:25:05.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.478+0000 7f019ffff640 10 client.4329.objecter ms_handle_connect 0x7f01a406df50 2024-03-25T19:25:05.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.480+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f01b00548d0 con 0x7f01b80b12d0 2024-03-25T19:25:05.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.480+0000 7f019ffff640 10 client.4329.objecter ms_dispatch 0x7f01b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:05.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.480+0000 7f019ffff640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:05.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.480+0000 7f019ffff640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:05.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.480+0000 7f019ffff640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-25T19:25:05.753 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.753+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01b0050210 con 0x7f01b80b12d0 2024-03-25T19:25:05.753 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.753+0000 7f019ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:05.753 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.753+0000 7f019ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:05.968 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.968+0000 7f01bd910640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:05.968 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.968+0000 7f01bd910640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:05.968 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.968+0000 7f01bd910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:25:05.968 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.968+0000 7f01bd910640 1 -- 172.21.15.123:0/3440908960 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01b81c5730 con 0x7f01b80b12d0 2024-03-25T19:25:05.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.989+0000 7f019ffff640 1 -- 172.21.15.123:0/3440908960 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f01b81c5730 con 0x7f01b80b12d0 2024-03-25T19:25:05.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.989+0000 7f019ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:05.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.989+0000 7f019ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:05.989 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:25:05.990 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:05.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.991+0000 7f01bd910640 10 librados: watch_flush enter 2024-03-25T19:25:05.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.991+0000 7f01bd910640 10 librados: watch_flush exit 2024-03-25T19:25:05.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.991+0000 7f01bd910640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-25T19:25:05.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.991+0000 7f01bd910640 10 client.4329.objecter successfully canceled tick 2024-03-25T19:25:05.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.991+0000 7f01bd910640 1 -- 172.21.15.123:0/3440908960 >> v1:172.21.15.151:6800/727684239 conn(0x7f01a406df50 legacy=0x7f01a4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.991+0000 7f01bd910640 10 monclient: shutdown 2024-03-25T19:25:05.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.991+0000 7f01bd910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:05.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.991+0000 7f01bd910640 1 -- 172.21.15.123:0/3440908960 >> v1:172.21.15.123:6789/0 conn(0x7f01b80b12d0 legacy=0x7f01b81dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.992+0000 7f01bd910640 1 -- 172.21.15.123:0/3440908960 shutdown_connections 2024-03-25T19:25:05.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.992+0000 7f01bd910640 1 -- 172.21.15.123:0/3440908960 >> 172.21.15.123:0/3440908960 conn(0x7f01b80b5d10 msgr2=0x7f01b81cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:05.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.992+0000 7f01bd910640 1 -- 172.21.15.123:0/3440908960 shutdown_connections 2024-03-25T19:25:05.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.992+0000 7f01bd910640 1 -- 172.21.15.123:0/3440908960 wait complete. 2024-03-25T19:25:05.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:05.992+0000 7f01bd910640 1 librados: shutdown 2024-03-25T19:25:06.004 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:09.953 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:02:59.525 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:09.962 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:02:59.534 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:09.998 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:02:59.571 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.000 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:02:59.573 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.007 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:02:59.540 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.014 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:02:59.586 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.014 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:02:59.548 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.042 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:02:59.614 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:10.044 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:02:59.578 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:12.809 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:12.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.925+0000 7fb71fb0e640 10 monclient: get_monmap_and_config 2024-03-25T19:25:12.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.925+0000 7fb71fb0e640 10 monclient: build_initial_monmap 2024-03-25T19:25:12.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.925+0000 7fb71fb0e640 10 monclient: monmap: 2024-03-25T19:25:12.925 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:12.925 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:12.925 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:25:12.925957+0000 2024-03-25T19:25:12.925 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:25:12.925957+0000 2024-03-25T19:25:12.925 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:12.925 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:12.925 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:12.925 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:25:12.926 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:25:12.926 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:12.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.925+0000 7fb71fb0e640 1 Processor -- start 2024-03-25T19:25:12.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 1 -- start start 2024-03-25T19:25:12.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 10 monclient: init 2024-03-25T19:25:12.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:12.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:12.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 10 monclient(hunting): picked mon.noname-c con 0x7fb7180a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:12.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 10 monclient(hunting): picked mon.noname-b con 0x7fb71813f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:12.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 10 monclient(hunting): picked mon.noname-a con 0x7fb7181430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:12.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb718142520 con 0x7fb7181430c0 2024-03-25T19:25:12.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7180b28a0 con 0x7fb7180a98f0 2024-03-25T19:25:12.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7180b2a30 con 0x7fb71813f710 2024-03-25T19:25:12.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:12.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.926+0000 7fb71fb0e640 10 monclient(hunting): authenticate will time out at 5750.286754s 2024-03-25T19:25:12.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.943+0000 7fb71d8aa640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fb7180a98f0 0x7fb7180a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:36688/0 (socket says 172.21.15.123:36688) 2024-03-25T19:25:12.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.943+0000 7fb71d8aa640 1 -- 172.21.15.123:0/1612790349 learned_addr learned my addr 172.21.15.123:0/1612790349 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:12.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.965+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448681545 0 0) 0x7fb7180b2a30 con 0x7fb71813f710 2024-03-25T19:25:12.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.965+0000 7fb71c8a8640 10 monclient(hunting): _negotiate 2024-03-25T19:25:12.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.965+0000 7fb71c8a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:12.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.965+0000 7fb71c8a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:12.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.965+0000 7fb71c8a8640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:25:12.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.965+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb704003520 con 0x7fb71813f710 2024-03-25T19:25:12.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.967+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808684392 0 0) 0x7fb718142520 con 0x7fb7181430c0 2024-03-25T19:25:12.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.967+0000 7fb71c8a8640 10 monclient(hunting): _negotiate 2024-03-25T19:25:12.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.967+0000 7fb71c8a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:12.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.967+0000 7fb71c8a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:12.968 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.967+0000 7fb71c8a8640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:25:12.968 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.967+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7180b2a30 con 0x7fb7181430c0 2024-03-25T19:25:12.969 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.969+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803508512 0 0) 0x7fb7180b28a0 con 0x7fb7180a98f0 2024-03-25T19:25:12.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.969+0000 7fb71c8a8640 10 monclient(hunting): _negotiate 2024-03-25T19:25:12.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.969+0000 7fb71c8a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:12.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.969+0000 7fb71c8a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:12.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.969+0000 7fb71c8a8640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:25:12.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.969+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb718142520 con 0x7fb7180a98f0 2024-03-25T19:25:12.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3279883166 0 0) 0x7fb7180b2a30 con 0x7fb7181430c0 2024-03-25T19:25:12.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 >> v1:172.21.15.123:6790/0 conn(0x7fb71813f710 legacy=0x7fb7181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 >> v1:172.21.15.151:6789/0 conn(0x7fb7180a98f0 legacy=0x7fb7180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 10 monclient: _finish_hunting 0 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 1 monclient: found mon.noname-a 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7180b44c0 con 0x7fb7181430c0 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 10 monclient: _finish_auth 0 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 10 monclient: _check_auth_tickets 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb7100019f0 con 0x7fb7181430c0 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71c8a8640 10 monclient: dump: 2024-03-25T19:25:12.984 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71fb0e640 5 monclient: authenticate success, global_id 4332 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:12.983+0000 7fb71fb0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:13.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.016+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb710001b50 con 0x7fb7181430c0 2024-03-25T19:25:13.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.016+0000 7fb71c8a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:13.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.016+0000 7fb71c8a8640 1 -- 172.21.15.123:0/1612790349 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb710001f60 con 0x7fb7181430c0 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.016+0000 7fb71fb0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.016+0000 7fb71fb0e640 4 set_mon_vals no callback set 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.016+0000 7fb71c8a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.016+0000 7fb71c8a8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.016+0000 7fb71c8a8640 10 monclient: dump: 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:13.017 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:13.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.017+0000 7fb71fb0e640 10 monclient: shutdown 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.017+0000 7fb71fb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.017+0000 7fb71fb0e640 1 -- 172.21.15.123:0/1612790349 >> v1:172.21.15.123:6789/0 conn(0x7fb7181430c0 legacy=0x7fb71814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.017+0000 7fb71fb0e640 1 -- 172.21.15.123:0/1612790349 shutdown_connections 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.017+0000 7fb71fb0e640 1 -- 172.21.15.123:0/1612790349 >> 172.21.15.123:0/1612790349 conn(0x7fb7180b7ea0 msgr2=0x7fb7180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.017+0000 7fb71fb0e640 1 -- 172.21.15.123:0/1612790349 shutdown_connections 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.017+0000 7fb71fb0e640 1 -- 172.21.15.123:0/1612790349 wait complete. 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.018+0000 7fb71fb0e640 10 monclient: build_initial_monmap 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.018+0000 7fb71fb0e640 10 monclient: monmap: 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:25:13.020 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.018+0000 7fb71fb0e640 1 librados: starting msgr at 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.018+0000 7fb71fb0e640 1 librados: starting objecter 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.018+0000 7fb71fb0e640 1 Processor -- start 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.018+0000 7fb71fb0e640 1 -- start start 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.018+0000 7fb71fb0e640 1 librados: setting wanted keys 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.018+0000 7fb71fb0e640 1 librados: calling monclient init 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.018+0000 7fb71fb0e640 10 monclient: init 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.019+0000 7fb71fb0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.019+0000 7fb71fb0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.019+0000 7fb71fb0e640 10 monclient(hunting): picked mon.noname-b con 0x7fb7180a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.019+0000 7fb71fb0e640 10 monclient(hunting): picked mon.noname-c con 0x7fb71813f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.019+0000 7fb71fb0e640 10 monclient(hunting): picked mon.noname-a con 0x7fb7181430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.019+0000 7fb71fb0e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb718138bd0 con 0x7fb7181430c0 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.019+0000 7fb71fb0e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7180aba30 con 0x7fb7180a98f0 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.019+0000 7fb71fb0e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb718137740 con 0x7fb71813f710 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.019+0000 7fb71fb0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.019+0000 7fb71fb0e640 10 monclient(hunting): authenticate will time out at 5750.379377s 2024-03-25T19:25:13.023 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.023+0000 7fb71d8aa640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fb7180a98f0 0x7fb7181384c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:36770/0 (socket says 172.21.15.123:36770) 2024-03-25T19:25:13.023 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.023+0000 7fb71d8aa640 1 -- 172.21.15.123:0/3642569894 learned_addr learned my addr 172.21.15.123:0/3642569894 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:13.023 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.024+0000 7fb70e7fc640 10 client.?.objecter ms_handle_connect 0x7fb7180a98f0 2024-03-25T19:25:13.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.024+0000 7fb70e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:13.034 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.034+0000 7fb70e7fc640 10 client.?.objecter ms_handle_connect 0x7fb7181430c0 2024-03-25T19:25:13.034 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.034+0000 7fb70e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:13.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.035+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407004982 0 0) 0x7fb7180aba30 con 0x7fb7180a98f0 2024-03-25T19:25:13.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.035+0000 7fb70e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:13.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.035+0000 7fb70e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:13.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.035+0000 7fb70e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:13.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.035+0000 7fb70e7fc640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:25:13.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.035+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6fc004eb0 con 0x7fb7180a98f0 2024-03-25T19:25:13.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.040+0000 7fb70e7fc640 10 client.?.objecter ms_handle_connect 0x7fb71813f710 2024-03-25T19:25:13.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.040+0000 7fb70e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:13.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.043+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3991079740 0 0) 0x7fb6fc004eb0 con 0x7fb7180a98f0 2024-03-25T19:25:13.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.043+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7180aba30 con 0x7fb7180a98f0 2024-03-25T19:25:13.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.043+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb714001910 con 0x7fb7180a98f0 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.043+0000 7fb70e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.043+0000 7fb70e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.043+0000 7fb70e7fc640 10 monclient(hunting): dump: 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:13.044 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:13.045 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:13.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.043+0000 7fb70e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:13.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.046+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171013181 0 0) 0x7fb718138bd0 con 0x7fb7181430c0 2024-03-25T19:25:13.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.046+0000 7fb70e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:13.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.046+0000 7fb70e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:13.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.046+0000 7fb70e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:13.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.046+0000 7fb70e7fc640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:25:13.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.046+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6fc004eb0 con 0x7fb7181430c0 2024-03-25T19:25:13.054 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.054+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733175668 0 0) 0x7fb718137740 con 0x7fb71813f710 2024-03-25T19:25:13.054 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.054+0000 7fb70e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:13.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.054+0000 7fb70e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:13.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.054+0000 7fb70e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:13.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.054+0000 7fb70e7fc640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:25:13.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.054+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb718138bd0 con 0x7fb71813f710 2024-03-25T19:25:13.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2317090235 0 0) 0x7fb7180aba30 con 0x7fb7180a98f0 2024-03-25T19:25:13.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 >> v1:172.21.15.123:6790/0 conn(0x7fb71813f710 legacy=0x7fb718134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:13.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 >> v1:172.21.15.123:6789/0 conn(0x7fb7181430c0 legacy=0x7fb7181d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:13.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 1 monclient: found mon.b 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb718134de0 con 0x7fb7180a98f0 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb70e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb71fb0e640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.057+0000 7fb71fb0e640 10 monclient: _renew_subs 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.058+0000 7fb71fb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.058+0000 7fb71fb0e640 1 -- 172.21.15.123:0/3642569894 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7180b1620 con 0x7fb7180a98f0 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.058+0000 7fb71fb0e640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:13.059 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.058+0000 7fb71fb0e640 10 monclient: _renew_subs 2024-03-25T19:25:13.060 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.058+0000 7fb71fb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:25:13.060 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.058+0000 7fb71fb0e640 1 -- 172.21.15.123:0/3642569894 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb718134900 con 0x7fb7180a98f0 2024-03-25T19:25:13.060 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.058+0000 7fb71fb0e640 1 librados: init done 2024-03-25T19:25:13.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.059+0000 7fb71fb0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:13.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.059+0000 7fb71fb0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:13.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.059+0000 7fb71fb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:25:13.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.059+0000 7fb71fb0e640 1 -- 172.21.15.123:0/3642569894 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7181d75d0 con 0x7fb7180a98f0 2024-03-25T19:25:13.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.061+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7140047e0 con 0x7fb7180a98f0 2024-03-25T19:25:13.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.061+0000 7fb70e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:13.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.061+0000 7fb70ffff640 4 set_mon_vals no callback set 2024-03-25T19:25:13.343 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.343+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb714004c30 con 0x7fb7180a98f0 2024-03-25T19:25:13.343 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.343+0000 7fb70e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:13.343 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.343+0000 7fb70e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:13.343 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.343+0000 7fb70e7fc640 10 monclient: dump: 2024-03-25T19:25:13.343 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:13.343 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:13.344 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:13.344 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:13.344 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:13.344 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:13.344 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:13.344 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:13.344 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:13.344 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:13.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.344+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fb714004e10 con 0x7fb7180a98f0 2024-03-25T19:25:13.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.344+0000 7fb71d0a9640 10 monclient: get_auth_request con 0x7fb6fc06de50 auth_method 0 2024-03-25T19:25:13.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.344+0000 7fb70e7fc640 10 client.4351.objecter ms_handle_connect 0x7fb6fc06de50 2024-03-25T19:25:13.347 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.347+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fb714054910 con 0x7fb7180a98f0 2024-03-25T19:25:13.347 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.347+0000 7fb70e7fc640 10 client.4351.objecter ms_dispatch 0x7fb718000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:13.347 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.347+0000 7fb70e7fc640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:13.347 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.347+0000 7fb70e7fc640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:13.347 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.347+0000 7fb70e7fc640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T19:25:13.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.350+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb714050420 con 0x7fb7180a98f0 2024-03-25T19:25:13.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.350+0000 7fb70e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:13.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.350+0000 7fb70e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:13.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.599+0000 7fb71fb0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:13.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.599+0000 7fb71fb0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:13.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.599+0000 7fb71fb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:25:13.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.599+0000 7fb71fb0e640 1 -- 172.21.15.123:0/3642569894 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7180627b0 con 0x7fb7180a98f0 2024-03-25T19:25:13.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.630+0000 7fb70e7fc640 1 -- 172.21.15.123:0/3642569894 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fb714052350 con 0x7fb7180a98f0 2024-03-25T19:25:13.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.630+0000 7fb70e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:13.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.630+0000 7fb70e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:13.630 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:25:13.630 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:13.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.632+0000 7fb71fb0e640 10 librados: watch_flush enter 2024-03-25T19:25:13.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.632+0000 7fb71fb0e640 10 librados: watch_flush exit 2024-03-25T19:25:13.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.632+0000 7fb71fb0e640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T19:25:13.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.632+0000 7fb71fb0e640 10 client.4351.objecter successfully canceled tick 2024-03-25T19:25:13.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.632+0000 7fb71fb0e640 1 -- 172.21.15.123:0/3642569894 >> v1:172.21.15.151:6800/727684239 conn(0x7fb6fc06de50 legacy=0x7fb6fc070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:13.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.633+0000 7fb71fb0e640 10 monclient: shutdown 2024-03-25T19:25:13.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.633+0000 7fb71fb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:13.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.633+0000 7fb71fb0e640 1 -- 172.21.15.123:0/3642569894 >> v1:172.21.15.151:6789/0 conn(0x7fb7180a98f0 legacy=0x7fb7181384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:13.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.633+0000 7fb71fb0e640 1 -- 172.21.15.123:0/3642569894 shutdown_connections 2024-03-25T19:25:13.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.633+0000 7fb71fb0e640 1 -- 172.21.15.123:0/3642569894 >> 172.21.15.123:0/3642569894 conn(0x7fb7180b7ea0 msgr2=0x7fb718074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:13.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.633+0000 7fb71fb0e640 1 -- 172.21.15.123:0/3642569894 shutdown_connections 2024-03-25T19:25:13.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.633+0000 7fb71fb0e640 1 -- 172.21.15.123:0/3642569894 wait complete. 2024-03-25T19:25:13.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:13.633+0000 7fb71fb0e640 1 librados: shutdown 2024-03-25T19:25:13.644 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:20.451 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:20.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.565+0000 7f2bac94d640 10 monclient: get_monmap_and_config 2024-03-25T19:25:20.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.565+0000 7f2bac94d640 10 monclient: build_initial_monmap 2024-03-25T19:25:20.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.565+0000 7f2bac94d640 10 monclient: monmap: 2024-03-25T19:25:20.565 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:20.565 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:20.565 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:25:20.566158+0000 2024-03-25T19:25:20.565 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:25:20.566158+0000 2024-03-25T19:25:20.566 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:20.566 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:20.566 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:20.566 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:25:20.566 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:25:20.566 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:20.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.566+0000 7f2bac94d640 1 Processor -- start 2024-03-25T19:25:20.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.566+0000 7f2bac94d640 1 -- start start 2024-03-25T19:25:20.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.566+0000 7f2bac94d640 10 monclient: init 2024-03-25T19:25:20.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.567+0000 7f2bac94d640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:20.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.567+0000 7f2bac94d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:25:20.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.567+0000 7f2bac94d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ba80b1ef0 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:20.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.567+0000 7f2bac94d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ba81473a0 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:20.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.567+0000 7f2bac94d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ba8150070 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:20.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.567+0000 7f2bac94d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba80b2570 con 0x7f2ba81473a0 2024-03-25T19:25:20.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.567+0000 7f2bac94d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba8146c20 con 0x7f2ba8150070 2024-03-25T19:25:20.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.567+0000 7f2bac94d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba8146e80 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.567+0000 7f2bac94d640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:20.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.567+0000 7f2bac94d640 10 monclient(hunting): authenticate will time out at 5757.927163s 2024-03-25T19:25:20.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.580+0000 7f2ba5d9b640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2ba81473a0 0x7f2ba8143b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46516/0 (socket says 172.21.15.123:46516) 2024-03-25T19:25:20.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.581+0000 7f2ba5d9b640 1 -- 172.21.15.123:0/1707270245 learned_addr learned my addr 172.21.15.123:0/1707270245 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:20.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.595+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636373615 0 0) 0x7f2ba8146e80 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.595+0000 7f2ba559a640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.595+0000 7f2ba559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.595+0000 7f2ba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:20.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.595+0000 7f2ba559a640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:25:20.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.595+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b90003520 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324916085 0 0) 0x7f2ba80b2570 con 0x7f2ba81473a0 2024-03-25T19:25:20.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:20.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:25:20.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba8146e80 con 0x7f2ba81473a0 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798587538 0 0) 0x7f2ba8146c20 con 0x7f2ba8150070 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.597+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba80b2570 con 0x7f2ba8150070 2024-03-25T19:25:20.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063723316 0 0) 0x7f2ba80b2570 con 0x7f2ba8150070 2024-03-25T19:25:20.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 >> v1:172.21.15.123:6790/0 conn(0x7f2ba80b1ef0 legacy=0x7f2ba80b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 >> v1:172.21.15.123:6789/0 conn(0x7f2ba81473a0 legacy=0x7f2ba8143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 10 monclient: _finish_hunting 0 2024-03-25T19:25:20.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 1 monclient: found mon.noname-c 2024-03-25T19:25:20.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:20.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:25:20.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba806e1b0 con 0x7f2ba8150070 2024-03-25T19:25:20.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 10 monclient: _finish_auth 0 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 10 monclient: _check_auth_tickets 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2ba0001930 con 0x7f2ba8150070 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2ba559a640 10 monclient: dump: 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:20.624 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:20.625 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:20.625 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:20.625 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:20.625 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:20.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2bac94d640 5 monclient: authenticate success, global_id 4354 2024-03-25T19:25:20.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.623+0000 7f2bac94d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:20.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.642+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ba0001a90 con 0x7f2ba8150070 2024-03-25T19:25:20.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.642+0000 7f2ba559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.642+0000 7f2ba559a640 1 -- 172.21.15.123:0/1707270245 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2ba0001ee0 con 0x7f2ba8150070 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.642+0000 7f2ba559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.642+0000 7f2ba559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.642+0000 7f2ba559a640 10 monclient: dump: 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:20.643 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:20.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.642+0000 7f2bac94d640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:20.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.642+0000 7f2bac94d640 4 set_mon_vals no callback set 2024-03-25T19:25:20.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.643+0000 7f2bac94d640 10 monclient: shutdown 2024-03-25T19:25:20.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.643+0000 7f2bac94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:20.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.643+0000 7f2bac94d640 1 -- 172.21.15.123:0/1707270245 >> v1:172.21.15.151:6789/0 conn(0x7f2ba8150070 legacy=0x7f2ba8152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.644+0000 7f2bac94d640 1 -- 172.21.15.123:0/1707270245 shutdown_connections 2024-03-25T19:25:20.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.644+0000 7f2bac94d640 1 -- 172.21.15.123:0/1707270245 >> 172.21.15.123:0/1707270245 conn(0x7f2ba80b7ea0 msgr2=0x7f2ba80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:20.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.644+0000 7f2bac94d640 1 -- 172.21.15.123:0/1707270245 shutdown_connections 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.644+0000 7f2bac94d640 1 -- 172.21.15.123:0/1707270245 wait complete. 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.645+0000 7f2bac94d640 10 monclient: build_initial_monmap 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.645+0000 7f2bac94d640 10 monclient: monmap: 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:20.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.645+0000 7f2bac94d640 1 librados: starting msgr at 2024-03-25T19:25:20.647 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.645+0000 7f2bac94d640 1 librados: starting objecter 2024-03-25T19:25:20.647 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.645+0000 7f2bac94d640 1 Processor -- start 2024-03-25T19:25:20.647 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.645+0000 7f2bac94d640 1 -- start start 2024-03-25T19:25:20.647 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.645+0000 7f2bac94d640 1 librados: setting wanted keys 2024-03-25T19:25:20.647 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.645+0000 7f2bac94d640 1 librados: calling monclient init 2024-03-25T19:25:20.647 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.645+0000 7f2bac94d640 10 monclient: init 2024-03-25T19:25:20.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.646+0000 7f2bac94d640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:20.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.646+0000 7f2bac94d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:25:20.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.646+0000 7f2bac94d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ba80b1ef0 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:20.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.646+0000 7f2bac94d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ba81473a0 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:20.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.646+0000 7f2bac94d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ba8150070 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:20.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.646+0000 7f2bac94d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba8140300 con 0x7f2ba8150070 2024-03-25T19:25:20.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.646+0000 7f2bac94d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba813f3c0 con 0x7f2ba81473a0 2024-03-25T19:25:20.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.646+0000 7f2bac94d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba813fef0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.646+0000 7f2bac94d640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:20.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.646+0000 7f2bac94d640 10 monclient(hunting): authenticate will time out at 5758.006114s 2024-03-25T19:25:20.649 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.650+0000 7f2ba659c640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f2ba80b1ef0 0x7f2ba81deb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:46994/0 (socket says 172.21.15.123:46994) 2024-03-25T19:25:20.649 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.650+0000 7f2ba659c640 1 -- 172.21.15.123:0/29176543 learned_addr learned my addr 172.21.15.123:0/29176543 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:20.650 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.650+0000 7f2b96ffd640 10 client.?.objecter ms_handle_connect 0x7f2ba80b1ef0 2024-03-25T19:25:20.650 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.650+0000 7f2b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:20.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.654+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652944264 0 0) 0x7f2ba813fef0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.654+0000 7f2b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.655 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.654+0000 7f2b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.655 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.654+0000 7f2b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:20.655 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.654+0000 7f2b96ffd640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:25:20.655 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.654+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b88004eb0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.663+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1674192687 0 0) 0x7f2b88004eb0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.663+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ba813fef0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.663+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2b980019b0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.663+0000 7f2b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.663+0000 7f2b96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.663+0000 7f2b96ffd640 10 monclient(hunting): dump: 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.663+0000 7f2b96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:20.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 829519413 0 0) 0x7f2ba813fef0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 >> v1:172.21.15.151:6789/0 conn(0x7f2ba81473a0 legacy=0x7f2ba81df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 >> v1:172.21.15.123:6789/0 conn(0x7f2ba8150070 legacy=0x7f2ba81e6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:25:20.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 1 monclient: found mon.c 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba8140790 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 10 monclient: _finish_auth 0 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2bac94d640 5 monclient: authenticate success, global_id 4364 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2bac94d640 10 monclient: _renew_subs 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2bac94d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2bac94d640 1 -- 172.21.15.123:0/29176543 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba813f5e0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2bac94d640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2bac94d640 10 monclient: _renew_subs 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2bac94d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:20.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2bac94d640 1 -- 172.21.15.123:0/29176543 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba81457e0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.672+0000 7f2bac94d640 1 librados: init done 2024-03-25T19:25:20.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.673+0000 7f2bac94d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:20.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.673+0000 7f2bac94d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:20.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.673+0000 7f2bac94d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:20.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.673+0000 7f2bac94d640 1 -- 172.21.15.123:0/29176543 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba80627b0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.964+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b98001b10 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.964+0000 7f2b96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.964+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2b98003a60 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.964+0000 7f2b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.964+0000 7f2b96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.964+0000 7f2b96ffd640 10 monclient: dump: 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:20.965 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:20.966 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:20.966 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:20.966 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:20.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.964+0000 7f2ba4d99640 4 set_mon_vals no callback set 2024-03-25T19:25:20.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.965+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f2b98003c80 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.966 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.966+0000 7f2ba5d9b640 10 monclient: get_auth_request con 0x7f2b8806dbc0 auth_method 0 2024-03-25T19:25:20.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.966+0000 7f2b96ffd640 10 client.4364.objecter ms_handle_connect 0x7f2b8806dbc0 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.973+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f2b980548c0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.973+0000 7f2b96ffd640 10 client.4364.objecter ms_dispatch 0x7f2ba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.973+0000 7f2b96ffd640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.973+0000 7f2b96ffd640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.973+0000 7f2b96ffd640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.973+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b980503d0 con 0x7f2ba80b1ef0 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.974+0000 7f2b96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:20.974+0000 7f2b96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:21.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.190+0000 7f2bac94d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:21.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.190+0000 7f2bac94d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:21.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.190+0000 7f2bac94d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:21.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.190+0000 7f2bac94d640 1 -- 172.21.15.123:0/29176543 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ba8140080 con 0x7f2ba80b1ef0 2024-03-25T19:25:21.292 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.292+0000 7f2b96ffd640 1 -- 172.21.15.123:0/29176543 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f2b98052300 con 0x7f2ba80b1ef0 2024-03-25T19:25:21.293 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.292+0000 7f2b96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:21.293 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.292+0000 7f2b96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:21.293 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:25:21.293 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:21.298 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.294+0000 7f2bac94d640 10 librados: watch_flush enter 2024-03-25T19:25:21.298 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.294+0000 7f2bac94d640 10 librados: watch_flush exit 2024-03-25T19:25:21.298 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.294+0000 7f2bac94d640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-25T19:25:21.298 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.294+0000 7f2bac94d640 10 client.4364.objecter successfully canceled tick 2024-03-25T19:25:21.298 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.294+0000 7f2bac94d640 1 -- 172.21.15.123:0/29176543 >> v1:172.21.15.151:6800/727684239 conn(0x7f2b8806dbc0 legacy=0x7f2b88070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:21.298 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.294+0000 7f2bac94d640 10 monclient: shutdown 2024-03-25T19:25:21.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.294+0000 7f2bac94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:21.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.294+0000 7f2bac94d640 1 -- 172.21.15.123:0/29176543 >> v1:172.21.15.123:6790/0 conn(0x7f2ba80b1ef0 legacy=0x7f2ba81deb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:21.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.294+0000 7f2bac94d640 1 -- 172.21.15.123:0/29176543 shutdown_connections 2024-03-25T19:25:21.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.294+0000 7f2bac94d640 1 -- 172.21.15.123:0/29176543 >> 172.21.15.123:0/29176543 conn(0x7f2ba80b7ea0 msgr2=0x7f2ba813d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:21.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.295+0000 7f2bac94d640 1 -- 172.21.15.123:0/29176543 shutdown_connections 2024-03-25T19:25:21.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.295+0000 7f2bac94d640 1 -- 172.21.15.123:0/29176543 wait complete. 2024-03-25T19:25:21.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:21.295+0000 7f2bac94d640 1 librados: shutdown 2024-03-25T19:25:21.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:28.114 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:28.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.230+0000 7f8237463640 10 monclient: get_monmap_and_config 2024-03-25T19:25:28.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.230+0000 7f8237463640 10 monclient: build_initial_monmap 2024-03-25T19:25:28.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.230+0000 7f8237463640 10 monclient: monmap: 2024-03-25T19:25:28.231 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:28.231 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:28.231 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:25:28.231712+0000 2024-03-25T19:25:28.231 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:25:28.231712+0000 2024-03-25T19:25:28.231 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:28.232 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:28.232 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:28.232 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:25:28.232 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:25:28.232 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:28.232 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.231+0000 7f8237463640 1 Processor -- start 2024-03-25T19:25:28.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.231+0000 7f8237463640 1 -- start start 2024-03-25T19:25:28.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.231+0000 7f8237463640 10 monclient: init 2024-03-25T19:25:28.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.232+0000 7f8237463640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:28.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.232+0000 7f8237463640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:28.233 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.232+0000 7f8237463640 10 monclient(hunting): picked mon.noname-a con 0x7f82300a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:28.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.232+0000 7f8237463640 10 monclient(hunting): picked mon.noname-c con 0x7f823013f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:28.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.232+0000 7f8237463640 10 monclient(hunting): picked mon.noname-b con 0x7f82301430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:28.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.232+0000 7f8237463640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8230142520 con 0x7f82300a98f0 2024-03-25T19:25:28.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.232+0000 7f8237463640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82300b28a0 con 0x7f823013f710 2024-03-25T19:25:28.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.232+0000 7f8237463640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82300b2a30 con 0x7f82301430c0 2024-03-25T19:25:28.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.232+0000 7f8237463640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:28.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.232+0000 7f8237463640 10 monclient(hunting): authenticate will time out at 5765.592572s 2024-03-25T19:25:28.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.250+0000 7f82349fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f823013f710 0x7f82301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:33276/0 (socket says 172.21.15.123:33276) 2024-03-25T19:25:28.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.250+0000 7f82349fe640 1 -- 172.21.15.123:0/4235709094 learned_addr learned my addr 172.21.15.123:0/4235709094 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:28.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.261+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092604591 0 0) 0x7f82300b28a0 con 0x7f823013f710 2024-03-25T19:25:28.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.261+0000 7f8227fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.262 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.261+0000 7f8227fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.262 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.261+0000 7f8227fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.262 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.261+0000 7f8227fff640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:25:28.262 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.261+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f821c003520 con 0x7f823013f710 2024-03-25T19:25:28.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.269+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593285636 0 0) 0x7f82300b2a30 con 0x7f82301430c0 2024-03-25T19:25:28.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.269+0000 7f8227fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.270 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.269+0000 7f8227fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.270 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.269+0000 7f8227fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.270 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.269+0000 7f8227fff640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:25:28.270 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.269+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82300b28a0 con 0x7f82301430c0 2024-03-25T19:25:28.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.273+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232535008 0 0) 0x7f8230142520 con 0x7f82300a98f0 2024-03-25T19:25:28.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.273+0000 7f8227fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.273+0000 7f8227fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.273+0000 7f8227fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.273+0000 7f8227fff640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:25:28.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.273+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82300b2a30 con 0x7f82300a98f0 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3587237986 0 0) 0x7f82300b2a30 con 0x7f82300a98f0 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 >> v1:172.21.15.123:6790/0 conn(0x7f82301430c0 legacy=0x7f823014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 >> v1:172.21.15.151:6789/0 conn(0x7f823013f710 legacy=0x7f82301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 10 monclient: _finish_hunting 0 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 1 monclient: found mon.noname-a 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82300b44c0 con 0x7f82300a98f0 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 10 monclient: _finish_auth 0 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 10 monclient: _check_auth_tickets 2024-03-25T19:25:28.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8227fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:28.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8237463640 5 monclient: authenticate success, global_id 4341 2024-03-25T19:25:28.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.280+0000 7f8237463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:28.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f822c0014f0 con 0x7f82300a98f0 2024-03-25T19:25:28.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8227fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8227fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8227fff640 10 monclient: dump: 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:28.307 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f822c0017c0 con 0x7f82300a98f0 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8227fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8227fff640 1 -- 172.21.15.123:0/4235709094 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f822c001b50 con 0x7f82300a98f0 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8237463640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8237463640 4 set_mon_vals no callback set 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8227fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8227fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.306+0000 7f8227fff640 10 monclient: dump: 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:28.308 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:28.309 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:28.309 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.307+0000 7f8237463640 10 monclient: shutdown 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.307+0000 7f8237463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.307+0000 7f8237463640 1 -- 172.21.15.123:0/4235709094 >> v1:172.21.15.123:6789/0 conn(0x7f82300a98f0 legacy=0x7f82300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.307+0000 7f8237463640 1 -- 172.21.15.123:0/4235709094 shutdown_connections 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.307+0000 7f8237463640 1 -- 172.21.15.123:0/4235709094 >> 172.21.15.123:0/4235709094 conn(0x7f82300b7ea0 msgr2=0x7f82300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.307+0000 7f8237463640 1 -- 172.21.15.123:0/4235709094 shutdown_connections 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.307+0000 7f8237463640 1 -- 172.21.15.123:0/4235709094 wait complete. 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.308+0000 7f8237463640 10 monclient: build_initial_monmap 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.308+0000 7f8237463640 10 monclient: monmap: 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:25:28.311 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 1 librados: starting msgr at 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 1 librados: starting objecter 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 1 Processor -- start 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 1 -- start start 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 1 librados: setting wanted keys 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 1 librados: calling monclient init 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 10 monclient: init 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 10 monclient(hunting): picked mon.noname-a con 0x7f82300a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 10 monclient(hunting): picked mon.noname-b con 0x7f823013f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:28.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 10 monclient(hunting): picked mon.noname-c con 0x7f82301430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:28.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8230139800 con 0x7f82300a98f0 2024-03-25T19:25:28.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82301d4c70 con 0x7f823013f710 2024-03-25T19:25:28.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82300b16d0 con 0x7f82301430c0 2024-03-25T19:25:28.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:28.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.309+0000 7f8237463640 10 monclient(hunting): authenticate will time out at 5765.669734s 2024-03-25T19:25:28.316 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.317+0000 7f82349fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f823013f710 0x7f82300afe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:33322/0 (socket says 172.21.15.123:33322) 2024-03-25T19:25:28.316 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.317+0000 7f82349fe640 1 -- 172.21.15.123:0/2892836162 learned_addr learned my addr 172.21.15.123:0/2892836162 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:28.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.317+0000 7f8225ffb640 10 client.?.objecter ms_handle_connect 0x7f823013f710 2024-03-25T19:25:28.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.317+0000 7f8225ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:28.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.319+0000 7f8225ffb640 10 client.?.objecter ms_handle_connect 0x7f82300a98f0 2024-03-25T19:25:28.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.319+0000 7f8225ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:28.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.327+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275761529 0 0) 0x7f82301d4c70 con 0x7f823013f710 2024-03-25T19:25:28.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.327+0000 7f8225ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.327+0000 7f8225ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.327+0000 7f8225ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.327+0000 7f8225ffb640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:25:28.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.327+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8214004eb0 con 0x7f823013f710 2024-03-25T19:25:28.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.329+0000 7f8225ffb640 10 client.?.objecter ms_handle_connect 0x7f82301430c0 2024-03-25T19:25:28.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.329+0000 7f8225ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:28.332 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.333+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547354327 0 0) 0x7f8230139800 con 0x7f82300a98f0 2024-03-25T19:25:28.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.333+0000 7f8225ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.333+0000 7f8225ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.333+0000 7f8225ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.333+0000 7f8225ffb640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:25:28.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.333+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82301d4c70 con 0x7f82300a98f0 2024-03-25T19:25:28.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.337+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029014959 0 0) 0x7f82300b16d0 con 0x7f82301430c0 2024-03-25T19:25:28.337 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.337+0000 7f8225ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.337 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.337+0000 7f8225ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.337 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.337+0000 7f8225ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.337 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.337+0000 7f8225ffb640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:25:28.337 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.337+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8230139800 con 0x7f82301430c0 2024-03-25T19:25:28.337 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.337+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3435640031 0 0) 0x7f8214004eb0 con 0x7f823013f710 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.337+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82300b16d0 con 0x7f823013f710 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.338+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8220001950 con 0x7f823013f710 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.338+0000 7f8225ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.338+0000 7f8225ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.338+0000 7f8225ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:28.338 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:28.339 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:28.339 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:28.339 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.338+0000 7f8225ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:28.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1599169242 0 0) 0x7f82300b16d0 con 0x7f823013f710 2024-03-25T19:25:28.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 >> v1:172.21.15.123:6790/0 conn(0x7f82301430c0 legacy=0x7f82301d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 >> v1:172.21.15.123:6789/0 conn(0x7f82300a98f0 legacy=0x7f82301d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:25:28.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 1 monclient: found mon.b 2024-03-25T19:25:28.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:28.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:25:28.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82300b0c30 con 0x7f823013f710 2024-03-25T19:25:28.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 10 monclient: _finish_auth 0 2024-03-25T19:25:28.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8225ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8237463640 5 monclient: authenticate success, global_id 4360 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8237463640 10 monclient: _renew_subs 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8237463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.347+0000 7f8237463640 1 -- 172.21.15.123:0/2892836162 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82300b0ee0 con 0x7f823013f710 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.348+0000 7f8237463640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.348+0000 7f8237463640 10 monclient: _renew_subs 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.348+0000 7f8237463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.348+0000 7f8237463640 1 -- 172.21.15.123:0/2892836162 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82300b0780 con 0x7f823013f710 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.348+0000 7f8237463640 1 librados: init done 2024-03-25T19:25:28.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.349+0000 7f8237463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:28.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.349+0000 7f8237463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:28.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.349+0000 7f8237463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:25:28.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.349+0000 7f8237463640 1 -- 172.21.15.123:0/2892836162 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82301d4e20 con 0x7f823013f710 2024-03-25T19:25:28.353 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.354+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82200047e0 con 0x7f823013f710 2024-03-25T19:25:28.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.354+0000 7f8225ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:28.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.354+0000 7f82277fe640 4 set_mon_vals no callback set 2024-03-25T19:25:28.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.395+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8220004950 con 0x7f823013f710 2024-03-25T19:25:28.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.395+0000 7f8225ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.395+0000 7f8225ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.395+0000 7f8225ffb640 10 monclient: dump: 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:28.395 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:28.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.632+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f822001a920 con 0x7f823013f710 2024-03-25T19:25:28.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.632+0000 7f82351ff640 10 monclient: get_auth_request con 0x7f821406dda0 auth_method 0 2024-03-25T19:25:28.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.632+0000 7f8225ffb640 10 client.4360.objecter ms_handle_connect 0x7f821406dda0 2024-03-25T19:25:28.635 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.635+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f8220054710 con 0x7f823013f710 2024-03-25T19:25:28.635 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.635+0000 7f8225ffb640 10 client.4360.objecter ms_dispatch 0x7f8230000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:28.635 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.635+0000 7f8225ffb640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:28.635 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.635+0000 7f8225ffb640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:28.635 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.635+0000 7f8225ffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T19:25:28.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.638+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8220050050 con 0x7f823013f710 2024-03-25T19:25:28.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.638+0000 7f8225ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:28.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.638+0000 7f8225ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:28.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.854+0000 7f8237463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:28.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.854+0000 7f8237463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:28.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.854+0000 7f8237463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:25:28.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:28.854+0000 7f8237463640 1 -- 172.21.15.123:0/2892836162 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82300627b0 con 0x7f823013f710 2024-03-25T19:25:29.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.037+0000 7f8225ffb640 1 -- 172.21.15.123:0/2892836162 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f8220052150 con 0x7f823013f710 2024-03-25T19:25:29.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.037+0000 7f8225ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:29.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.037+0000 7f8225ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:29.037 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:25:29.037 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:29.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 10 librados: watch_flush enter 2024-03-25T19:25:29.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 10 librados: watch_flush exit 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 10 client.4360.objecter successfully canceled tick 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 1 -- 172.21.15.123:0/2892836162 >> v1:172.21.15.151:6800/727684239 conn(0x7f821406dda0 legacy=0x7f8214070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 10 monclient: shutdown 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 1 -- 172.21.15.123:0/2892836162 >> v1:172.21.15.151:6789/0 conn(0x7f823013f710 legacy=0x7f82300afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 1 -- 172.21.15.123:0/2892836162 shutdown_connections 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 1 -- 172.21.15.123:0/2892836162 >> 172.21.15.123:0/2892836162 conn(0x7f82300b7ea0 msgr2=0x7f8230074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 1 -- 172.21.15.123:0/2892836162 shutdown_connections 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 1 -- 172.21.15.123:0/2892836162 wait complete. 2024-03-25T19:25:29.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:29.039+0000 7f8237463640 1 librados: shutdown 2024-03-25T19:25:29.051 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:30.017 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:03:19.589 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:30.023 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:03:19.595 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:30.048 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:03:19.620 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:30.081 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:03:19.614 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:30.086 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:03:19.658 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:30.089 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:03:19.622 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:30.097 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:03:19.669 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:30.111 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:03:19.645 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:30.118 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:03:19.690 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:35.858 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:35.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.973+0000 7fa1c02ae640 10 monclient: get_monmap_and_config 2024-03-25T19:25:35.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.973+0000 7fa1c02ae640 10 monclient: build_initial_monmap 2024-03-25T19:25:35.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.973+0000 7fa1c02ae640 10 monclient: monmap: 2024-03-25T19:25:35.973 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:35.973 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:35.973 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:25:35.974628+0000 2024-03-25T19:25:35.973 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:25:35.974628+0000 2024-03-25T19:25:35.974 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:35.974 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:35.974 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:35.974 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:25:35.974 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:25:35.974 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:35.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.974+0000 7fa1c02ae640 1 Processor -- start 2024-03-25T19:25:35.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.974+0000 7fa1c02ae640 1 -- start start 2024-03-25T19:25:35.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.974+0000 7fa1c02ae640 10 monclient: init 2024-03-25T19:25:35.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.975+0000 7fa1c02ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:35.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.975+0000 7fa1c02ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:35.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.975+0000 7fa1c02ae640 10 monclient(hunting): picked mon.noname-c con 0x7fa1b80a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:35.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.975+0000 7fa1c02ae640 10 monclient(hunting): picked mon.noname-b con 0x7fa1b813f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:35.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.975+0000 7fa1c02ae640 10 monclient(hunting): picked mon.noname-a con 0x7fa1b81430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:35.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.975+0000 7fa1c02ae640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b8142520 con 0x7fa1b81430c0 2024-03-25T19:25:35.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.975+0000 7fa1c02ae640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b80b28a0 con 0x7fa1b80a98f0 2024-03-25T19:25:35.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.975+0000 7fa1c02ae640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b80b2a30 con 0x7fa1b813f710 2024-03-25T19:25:35.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.975+0000 7fa1c02ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:35.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.975+0000 7fa1c02ae640 10 monclient(hunting): authenticate will time out at 5773.335245s 2024-03-25T19:25:35.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.977+0000 7fa1be04a640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fa1b80a98f0 0x7fa1b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:42736/0 (socket says 172.21.15.123:42736) 2024-03-25T19:25:35.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.977+0000 7fa1be04a640 1 -- 172.21.15.123:0/150620831 learned_addr learned my addr 172.21.15.123:0/150620831 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:35.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.984+0000 7fa1bd048640 1 -- 172.21.15.123:0/150620831 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880775372 0 0) 0x7fa1b80b28a0 con 0x7fa1b80a98f0 2024-03-25T19:25:35.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.984+0000 7fa1bd048640 10 monclient(hunting): _negotiate 2024-03-25T19:25:35.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.984+0000 7fa1bd048640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:35.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.984+0000 7fa1bd048640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:35.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.984+0000 7fa1bd048640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:25:35.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.984+0000 7fa1bd048640 1 -- 172.21.15.123:0/150620831 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a4003520 con 0x7fa1b80a98f0 2024-03-25T19:25:35.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 1 -- 172.21.15.123:0/150620831 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4206016039 0 0) 0x7fa1a4003520 con 0x7fa1b80a98f0 2024-03-25T19:25:35.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 1 -- 172.21.15.123:0/150620831 >> v1:172.21.15.123:6790/0 conn(0x7fa1b813f710 legacy=0x7fa1b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:35.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 1 -- 172.21.15.123:0/150620831 >> v1:172.21.15.123:6789/0 conn(0x7fa1b81430c0 legacy=0x7fa1b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:25:35.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 10 monclient: _finish_hunting 0 2024-03-25T19:25:35.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 1 monclient: found mon.noname-c 2024-03-25T19:25:35.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 1 -- 172.21.15.123:0/150620831 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b80b44c0 con 0x7fa1b80a98f0 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 10 monclient: _finish_auth 0 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 10 monclient: _check_auth_tickets 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 1 -- 172.21.15.123:0/150620831 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa1b4001690 con 0x7fa1b80a98f0 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1bd048640 10 monclient: dump: 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:36.000 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:36.001 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:36.001 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:36.001 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:36.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1c02ae640 5 monclient: authenticate success, global_id 4363 2024-03-25T19:25:36.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:35.999+0000 7fa1c02ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:36.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.006+0000 7fa1bd048640 1 -- 172.21.15.123:0/150620831 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1b4001860 con 0x7fa1b80a98f0 2024-03-25T19:25:36.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.006+0000 7fa1bd048640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:36.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.006+0000 7fa1bd048640 1 -- 172.21.15.123:0/150620831 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa1b4001cb0 con 0x7fa1b80a98f0 2024-03-25T19:25:36.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.006+0000 7fa1bd048640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:36.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.006+0000 7fa1bd048640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.006+0000 7fa1bd048640 10 monclient: dump: 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.006+0000 7fa1c02ae640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:36.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.006+0000 7fa1c02ae640 4 set_mon_vals no callback set 2024-03-25T19:25:36.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.007+0000 7fa1c02ae640 10 monclient: shutdown 2024-03-25T19:25:36.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.007+0000 7fa1c02ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:36.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.007+0000 7fa1c02ae640 1 -- 172.21.15.123:0/150620831 >> v1:172.21.15.151:6789/0 conn(0x7fa1b80a98f0 legacy=0x7fa1b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:36.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.007+0000 7fa1c02ae640 1 -- 172.21.15.123:0/150620831 shutdown_connections 2024-03-25T19:25:36.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.007+0000 7fa1c02ae640 1 -- 172.21.15.123:0/150620831 >> 172.21.15.123:0/150620831 conn(0x7fa1b80b7ea0 msgr2=0x7fa1b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:36.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.007+0000 7fa1c02ae640 1 -- 172.21.15.123:0/150620831 shutdown_connections 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.007+0000 7fa1c02ae640 1 -- 172.21.15.123:0/150620831 wait complete. 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.008+0000 7fa1c02ae640 10 monclient: build_initial_monmap 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.008+0000 7fa1c02ae640 10 monclient: monmap: 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.008+0000 7fa1c02ae640 1 librados: starting msgr at 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.008+0000 7fa1c02ae640 1 librados: starting objecter 2024-03-25T19:25:36.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 1 Processor -- start 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 1 -- start start 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 1 librados: setting wanted keys 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 1 librados: calling monclient init 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 10 monclient: init 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 10 monclient(hunting): picked mon.noname-a con 0x7fa1b80a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 10 monclient(hunting): picked mon.noname-b con 0x7fa1b813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 10 monclient(hunting): picked mon.noname-c con 0x7fa1b81430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b8139a30 con 0x7fa1b80a98f0 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b81d7490 con 0x7fa1b813f710 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b81d8360 con 0x7fa1b81430c0 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:36.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.009+0000 7fa1c02ae640 10 monclient(hunting): authenticate will time out at 5773.369687s 2024-03-25T19:25:36.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.020+0000 7fa1bd849640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fa1b813f710 0x7fa1b81d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:42766/0 (socket says 172.21.15.123:42766) 2024-03-25T19:25:36.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.020+0000 7fa1bd849640 1 -- 172.21.15.123:0/2202948867 learned_addr learned my addr 172.21.15.123:0/2202948867 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:36.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.021+0000 7fa1aeffd640 10 client.?.objecter ms_handle_connect 0x7fa1b813f710 2024-03-25T19:25:36.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.021+0000 7fa1aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:36.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.027+0000 7fa1aeffd640 10 client.?.objecter ms_handle_connect 0x7fa1b81430c0 2024-03-25T19:25:36.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.027+0000 7fa1aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:36.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.029+0000 7fa1aeffd640 10 client.?.objecter ms_handle_connect 0x7fa1b80a98f0 2024-03-25T19:25:36.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.029+0000 7fa1aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:36.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.036+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266561883 0 0) 0x7fa1b81d7490 con 0x7fa1b813f710 2024-03-25T19:25:36.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.036+0000 7fa1aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:36.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.036+0000 7fa1aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:36.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.036+0000 7fa1aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:36.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.036+0000 7fa1aeffd640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:25:36.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.036+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa19c004eb0 con 0x7fa1b813f710 2024-03-25T19:25:36.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.039+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659634118 0 0) 0x7fa1b81d8360 con 0x7fa1b81430c0 2024-03-25T19:25:36.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.039+0000 7fa1aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:36.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.039+0000 7fa1aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:36.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.039+0000 7fa1aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:36.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.039+0000 7fa1aeffd640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:25:36.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.039+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b81d7490 con 0x7fa1b81430c0 2024-03-25T19:25:36.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.048+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400070122 0 0) 0x7fa1b8139a30 con 0x7fa1b80a98f0 2024-03-25T19:25:36.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.048+0000 7fa1aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:36.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.048+0000 7fa1aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:36.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.048+0000 7fa1aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:36.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.048+0000 7fa1aeffd640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:25:36.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.048+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b81d8360 con 0x7fa1b80a98f0 2024-03-25T19:25:36.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.052+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2467183328 0 0) 0x7fa19c004eb0 con 0x7fa1b813f710 2024-03-25T19:25:36.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.052+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1b8139a30 con 0x7fa1b813f710 2024-03-25T19:25:36.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.052+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa1a8001a30 con 0x7fa1b813f710 2024-03-25T19:25:36.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.052+0000 7fa1aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:36.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.052+0000 7fa1aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:36.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.052+0000 7fa1aeffd640 10 monclient(hunting): dump: 2024-03-25T19:25:36.052 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:36.053 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:36.053 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.053 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.053 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:36.053 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:36.053 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:36.053 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:36.053 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:36.053 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:36.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.052+0000 7fa1aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:36.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.055+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2601827191 0 0) 0x7fa1b81d7490 con 0x7fa1b81430c0 2024-03-25T19:25:36.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.055+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa19c004eb0 con 0x7fa1b81430c0 2024-03-25T19:25:36.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.055+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa1b0001a70 con 0x7fa1b81430c0 2024-03-25T19:25:36.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.055+0000 7fa1aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:36.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.055+0000 7fa1aeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:36.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.055+0000 7fa1aeffd640 10 monclient(hunting): dump: 2024-03-25T19:25:36.055 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:36.055 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:36.055 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.055 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.056 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:36.056 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:36.056 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:36.056 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:36.056 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:36.056 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:36.068 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.068+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3960972421 0 0) 0x7fa1b81d8360 con 0x7fa1b80a98f0 2024-03-25T19:25:36.068 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.068+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1b81d7490 con 0x7fa1b80a98f0 2024-03-25T19:25:36.068 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.068+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa1b40019d0 con 0x7fa1b80a98f0 2024-03-25T19:25:36.068 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.068+0000 7fa1aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:36.068 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.068+0000 7fa1aeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:36.068 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.068+0000 7fa1aeffd640 10 monclient(hunting): dump: 2024-03-25T19:25:36.069 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:36.069 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:36.069 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.069 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.069 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:36.069 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:36.069 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:36.069 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:36.069 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:36.069 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:36.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2826950784 0 0) 0x7fa19c004eb0 con 0x7fa1b81430c0 2024-03-25T19:25:36.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 >> v1:172.21.15.151:6789/0 conn(0x7fa1b813f710 legacy=0x7fa1b81d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:36.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 >> v1:172.21.15.123:6789/0 conn(0x7fa1b80a98f0 legacy=0x7fa1b8138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 1 monclient: found mon.c 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b81d7730 con 0x7fa1b81430c0 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1c02ae640 5 monclient: authenticate success, global_id 4376 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1c02ae640 10 monclient: _renew_subs 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1c02ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1c02ae640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b8134720 con 0x7fa1b81430c0 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1c02ae640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:36.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1c02ae640 10 monclient: _renew_subs 2024-03-25T19:25:36.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1c02ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:36.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1c02ae640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b8137a90 con 0x7fa1b81430c0 2024-03-25T19:25:36.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.071+0000 7fa1c02ae640 1 librados: init done 2024-03-25T19:25:36.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.072+0000 7fa1c02ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:36.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.072+0000 7fa1c02ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:36.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.072+0000 7fa1c02ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:36.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.072+0000 7fa1c02ae640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b8134a20 con 0x7fa1b81430c0 2024-03-25T19:25:36.374 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.374+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1b0001f60 con 0x7fa1b81430c0 2024-03-25T19:25:36.374 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.374+0000 7fa1aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:36.374 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.374+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa1b0003b10 con 0x7fa1b81430c0 2024-03-25T19:25:36.374 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.374+0000 7fa1aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:36.374 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.374+0000 7fa1aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.374+0000 7fa1aeffd640 10 monclient: dump: 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:36.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.374+0000 7fa1bc847640 4 set_mon_vals no callback set 2024-03-25T19:25:36.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.375+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fa1b0019ae0 con 0x7fa1b81430c0 2024-03-25T19:25:36.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.375+0000 7fa1be04a640 10 monclient: get_auth_request con 0x7fa19c06e160 auth_method 0 2024-03-25T19:25:36.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.375+0000 7fa1aeffd640 10 client.4376.objecter ms_handle_connect 0x7fa19c06e160 2024-03-25T19:25:36.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.380+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fa1b00536d0 con 0x7fa1b81430c0 2024-03-25T19:25:36.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.380+0000 7fa1aeffd640 10 client.4376.objecter ms_dispatch 0x7fa1b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:36.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.380+0000 7fa1aeffd640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:36.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.380+0000 7fa1aeffd640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:36.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.380+0000 7fa1aeffd640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-25T19:25:36.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.380+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1b004f1e0 con 0x7fa1b81430c0 2024-03-25T19:25:36.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.380+0000 7fa1aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:36.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.380+0000 7fa1aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:36.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.633+0000 7fa1c02ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:36.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.633+0000 7fa1c02ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:36.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.633+0000 7fa1c02ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:36.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.633+0000 7fa1c02ae640 1 -- 172.21.15.123:0/2202948867 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1b81d78c0 con 0x7fa1b81430c0 2024-03-25T19:25:36.664 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.664+0000 7fa1aeffd640 1 -- 172.21.15.123:0/2202948867 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fa1b0051110 con 0x7fa1b81430c0 2024-03-25T19:25:36.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.665+0000 7fa1aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:36.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.665+0000 7fa1aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:36.665 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:25:36.665 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:36.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 10 librados: watch_flush enter 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 10 librados: watch_flush exit 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 10 client.4376.objecter successfully canceled tick 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 1 -- 172.21.15.123:0/2202948867 >> v1:172.21.15.151:6800/727684239 conn(0x7fa19c06e160 legacy=0x7fa19c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 10 monclient: shutdown 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 1 -- 172.21.15.123:0/2202948867 >> v1:172.21.15.123:6790/0 conn(0x7fa1b81430c0 legacy=0x7fa1b8134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 1 -- 172.21.15.123:0/2202948867 shutdown_connections 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 1 -- 172.21.15.123:0/2202948867 >> 172.21.15.123:0/2202948867 conn(0x7fa1b80b7ea0 msgr2=0x7fa1b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 1 -- 172.21.15.123:0/2202948867 shutdown_connections 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 1 -- 172.21.15.123:0/2202948867 wait complete. 2024-03-25T19:25:36.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:36.667+0000 7fa1c02ae640 1 librados: shutdown 2024-03-25T19:25:36.678 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:43.485 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:43.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.599+0000 7f007498f640 10 monclient: get_monmap_and_config 2024-03-25T19:25:43.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.599+0000 7f007498f640 10 monclient: build_initial_monmap 2024-03-25T19:25:43.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.599+0000 7f007498f640 10 monclient: monmap: 2024-03-25T19:25:43.600 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:43.600 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:43.600 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:25:43.600553+0000 2024-03-25T19:25:43.600 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:25:43.600553+0000 2024-03-25T19:25:43.600 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:43.600 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:43.600 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:43.600 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:25:43.600 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:25:43.600 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.600+0000 7f007498f640 1 Processor -- start 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.600+0000 7f007498f640 1 -- start start 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.600+0000 7f007498f640 10 monclient: init 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.601+0000 7f007498f640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.601+0000 7f007498f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.601+0000 7f007498f640 10 monclient(hunting): picked mon.noname-a con 0x7f00700b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.601+0000 7f007498f640 10 monclient(hunting): picked mon.noname-c con 0x7f00700b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.601+0000 7f007498f640 10 monclient(hunting): picked mon.noname-b con 0x7f007014be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.601+0000 7f007498f640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00700af940 con 0x7f00700b4d60 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.601+0000 7f007498f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00700b0a70 con 0x7f00700b1730 2024-03-25T19:25:43.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.601+0000 7f007498f640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00700b0d50 con 0x7f007014be90 2024-03-25T19:25:43.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.601+0000 7f007498f640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:43.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.601+0000 7f007498f640 10 monclient(hunting): authenticate will time out at 5780.961381s 2024-03-25T19:25:43.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.613+0000 7f006ed9d640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f007014be90 0x7f007014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:47672/0 (socket says 172.21.15.123:47672) 2024-03-25T19:25:43.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.613+0000 7f006ed9d640 1 -- 172.21.15.123:0/1190605290 learned_addr learned my addr 172.21.15.123:0/1190605290 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:43.621 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.621+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330065567 0 0) 0x7f00700b0d50 con 0x7f007014be90 2024-03-25T19:25:43.621 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.621+0000 7f006d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:25:43.621 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.621+0000 7f006d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:43.621 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.621+0000 7f006d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:43.621 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.621+0000 7f006d59a640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:25:43.621 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.621+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0058003520 con 0x7f007014be90 2024-03-25T19:25:43.626 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.626+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017261253 0 0) 0x7f00700af940 con 0x7f00700b4d60 2024-03-25T19:25:43.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.626+0000 7f006d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:25:43.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.626+0000 7f006d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:43.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.626+0000 7f006d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:43.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.626+0000 7f006d59a640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:25:43.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.626+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00700b0d50 con 0x7f00700b4d60 2024-03-25T19:25:43.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779292215 0 0) 0x7f00700b0a70 con 0x7f00700b1730 2024-03-25T19:25:43.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:25:43.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:43.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00700af940 con 0x7f00700b1730 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 511490119 0 0) 0x7f00700b0d50 con 0x7f00700b4d60 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 >> v1:172.21.15.123:6790/0 conn(0x7f007014be90 legacy=0x7f007014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 >> v1:172.21.15.151:6789/0 conn(0x7f00700b1730 legacy=0x7f00700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 1 monclient: found mon.noname-a 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00700b1030 con 0x7f00700b4d60 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient: _finish_auth 0 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:43.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0060001670 con 0x7f00700b4d60 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f007498f640 5 monclient: authenticate success, global_id 4350 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f006d59a640 10 monclient: dump: 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:43.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.643+0000 7f007498f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:43.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00600017d0 con 0x7f00700b4d60 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f006d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f006d59a640 1 -- 172.21.15.123:0/1190605290 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0060001be0 con 0x7f00700b4d60 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f006d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f006d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f006d59a640 10 monclient: dump: 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:43.662 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:43.663 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:43.663 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:43.663 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:43.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f007498f640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:43.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f007498f640 4 set_mon_vals no callback set 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f007498f640 10 monclient: shutdown 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f007498f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.662+0000 7f007498f640 1 -- 172.21.15.123:0/1190605290 >> v1:172.21.15.123:6789/0 conn(0x7f00700b4d60 legacy=0x7f007013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.663+0000 7f007498f640 1 -- 172.21.15.123:0/1190605290 shutdown_connections 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.663+0000 7f007498f640 1 -- 172.21.15.123:0/1190605290 >> 172.21.15.123:0/1190605290 conn(0x7f00700617a0 msgr2=0x7f0070061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.663+0000 7f007498f640 1 -- 172.21.15.123:0/1190605290 shutdown_connections 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.663+0000 7f007498f640 1 -- 172.21.15.123:0/1190605290 wait complete. 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.663+0000 7f007498f640 10 monclient: build_initial_monmap 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.664+0000 7f007498f640 10 monclient: monmap: 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:43.665 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.664+0000 7f007498f640 1 librados: starting msgr at 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.664+0000 7f007498f640 1 librados: starting objecter 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.664+0000 7f007498f640 1 Processor -- start 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.664+0000 7f007498f640 1 -- start start 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.664+0000 7f007498f640 1 librados: setting wanted keys 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.664+0000 7f007498f640 1 librados: calling monclient init 2024-03-25T19:25:43.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.664+0000 7f007498f640 10 monclient: init 2024-03-25T19:25:43.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.665+0000 7f007498f640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:43.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.665+0000 7f007498f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:43.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.665+0000 7f007498f640 10 monclient(hunting): picked mon.noname-a con 0x7f00700b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:43.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.665+0000 7f007498f640 10 monclient(hunting): picked mon.noname-b con 0x7f00700b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:43.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.665+0000 7f007498f640 10 monclient(hunting): picked mon.noname-c con 0x7f007014be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:43.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.665+0000 7f007498f640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0070138500 con 0x7f00700b1730 2024-03-25T19:25:43.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.665+0000 7f007498f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00701db4b0 con 0x7f00700b4d60 2024-03-25T19:25:43.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.665+0000 7f007498f640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0070138920 con 0x7f007014be90 2024-03-25T19:25:43.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.665+0000 7f007498f640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:43.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.665+0000 7f007498f640 10 monclient(hunting): authenticate will time out at 5781.025119s 2024-03-25T19:25:43.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.683+0000 7f006e59c640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f00700b1730 0x7f00701dada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:43996/0 (socket says 172.21.15.123:43996) 2024-03-25T19:25:43.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.683+0000 7f006e59c640 1 -- 172.21.15.123:0/2532068908 learned_addr learned my addr 172.21.15.123:0/2532068908 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:43.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.684+0000 7f005effd640 10 client.?.objecter ms_handle_connect 0x7f00700b1730 2024-03-25T19:25:43.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.684+0000 7f005effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:43.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.684+0000 7f005effd640 10 client.?.objecter ms_handle_connect 0x7f00700b4d60 2024-03-25T19:25:43.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.684+0000 7f005effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:43.690 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.691+0000 7f005effd640 10 client.?.objecter ms_handle_connect 0x7f007014be90 2024-03-25T19:25:43.690 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.691+0000 7f005effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:43.696 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.696+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603968724 0 0) 0x7f00701db4b0 con 0x7f00700b4d60 2024-03-25T19:25:43.696 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.696+0000 7f005effd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:43.696 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.696+0000 7f005effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:43.696 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.696+0000 7f005effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:43.696 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.696+0000 7f005effd640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:25:43.696 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.696+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0050004eb0 con 0x7f00700b4d60 2024-03-25T19:25:43.698 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.698+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534354123 0 0) 0x7f0070138500 con 0x7f00700b1730 2024-03-25T19:25:43.698 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.698+0000 7f005effd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:43.698 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.698+0000 7f005effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:43.698 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.698+0000 7f005effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:43.699 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.698+0000 7f005effd640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:25:43.699 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.698+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00701db4b0 con 0x7f00700b1730 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.711+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872661661 0 0) 0x7f00701db4b0 con 0x7f00700b1730 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.711+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0070138500 con 0x7f00700b1730 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.711+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0060001ae0 con 0x7f00700b1730 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.711+0000 7f005effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.711+0000 7f005effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.711+0000 7f005effd640 10 monclient(hunting): dump: 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:43.712 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.712+0000 7f005effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.712+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610025235 0 0) 0x7f0070138920 con 0x7f007014be90 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.712+0000 7f005effd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.712+0000 7f005effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.712+0000 7f005effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.712+0000 7f005effd640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:25:43.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.712+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00701db4b0 con 0x7f007014be90 2024-03-25T19:25:43.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.717+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3333440621 0 0) 0x7f0050004eb0 con 0x7f00700b4d60 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.717+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0070138920 con 0x7f00700b4d60 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.717+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0064001970 con 0x7f00700b4d60 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.717+0000 7f005effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.717+0000 7f005effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.717+0000 7f005effd640 10 monclient(hunting): dump: 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:43.718 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:43.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3590229632 0 0) 0x7f0070138500 con 0x7f00700b1730 2024-03-25T19:25:43.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 >> v1:172.21.15.123:6790/0 conn(0x7f007014be90 legacy=0x7f00701e3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:43.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 >> v1:172.21.15.151:6789/0 conn(0x7f00700b4d60 legacy=0x7f00701370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:43.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 10 monclient: _finish_hunting 0 2024-03-25T19:25:43.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 1 monclient: found mon.a 2024-03-25T19:25:43.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:43.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:25:43.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0070138e60 con 0x7f00700b1730 2024-03-25T19:25:43.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 10 monclient: _finish_auth 0 2024-03-25T19:25:43.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 10 monclient: _check_auth_tickets 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f005effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f007498f640 5 monclient: authenticate success, global_id 4353 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f007498f640 10 monclient: _renew_subs 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f007498f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f007498f640 1 -- 172.21.15.123:0/2532068908 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0070139160 con 0x7f00700b1730 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f007498f640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f007498f640 10 monclient: _renew_subs 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f007498f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f007498f640 1 -- 172.21.15.123:0/2532068908 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f007013b3d0 con 0x7f00700b1730 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.727+0000 7f007498f640 1 librados: init done 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.728+0000 7f007498f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.728+0000 7f007498f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.728+0000 7f007498f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:25:43.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:43.728+0000 7f007498f640 1 -- 172.21.15.123:0/2532068908 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00701382d0 con 0x7f00700b1730 2024-03-25T19:25:44.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.047+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00600011f0 con 0x7f00700b1730 2024-03-25T19:25:44.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.047+0000 7f005effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:44.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.047+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0060003a40 con 0x7f00700b1730 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.047+0000 7f005effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.047+0000 7f005effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.047+0000 7f005effd640 10 monclient: dump: 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:44.048 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:44.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.047+0000 7f006cd99640 4 set_mon_vals no callback set 2024-03-25T19:25:44.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.048+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f0060003c60 con 0x7f00700b1730 2024-03-25T19:25:44.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.049+0000 7f006dd9b640 10 monclient: get_auth_request con 0x7f005006dfb0 auth_method 0 2024-03-25T19:25:44.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.049+0000 7f005effd640 10 client.4353.objecter ms_handle_connect 0x7f005006dfb0 2024-03-25T19:25:44.054 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.054+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f0060054880 con 0x7f00700b1730 2024-03-25T19:25:44.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.054+0000 7f005effd640 10 client.4353.objecter ms_dispatch 0x7f0070000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:44.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.054+0000 7f005effd640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:44.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.054+0000 7f005effd640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:44.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.054+0000 7f005effd640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T19:25:44.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.054+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00600501c0 con 0x7f00700b1730 2024-03-25T19:25:44.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.055+0000 7f005effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:44.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.055+0000 7f005effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:44.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.304+0000 7f007498f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:44.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.304+0000 7f007498f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:44.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.304+0000 7f007498f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:25:44.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.304+0000 7f007498f640 1 -- 172.21.15.123:0/2532068908 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00700627b0 con 0x7f00700b1730 2024-03-25T19:25:44.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.333+0000 7f005effd640 1 -- 172.21.15.123:0/2532068908 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f00600522c0 con 0x7f00700b1730 2024-03-25T19:25:44.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.333+0000 7f005effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:44.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.333+0000 7f005effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:44.333 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:25:44.333 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:44.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 10 librados: watch_flush enter 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 10 librados: watch_flush exit 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 10 client.4353.objecter successfully canceled tick 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 1 -- 172.21.15.123:0/2532068908 >> v1:172.21.15.151:6800/727684239 conn(0x7f005006dfb0 legacy=0x7f0050070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 10 monclient: shutdown 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 1 -- 172.21.15.123:0/2532068908 >> v1:172.21.15.123:6789/0 conn(0x7f00700b1730 legacy=0x7f00701dada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 1 -- 172.21.15.123:0/2532068908 shutdown_connections 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 1 -- 172.21.15.123:0/2532068908 >> 172.21.15.123:0/2532068908 conn(0x7f00700617a0 msgr2=0x7f00700b63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 1 -- 172.21.15.123:0/2532068908 shutdown_connections 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 1 -- 172.21.15.123:0/2532068908 wait complete. 2024-03-25T19:25:44.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:44.335+0000 7f007498f640 1 librados: shutdown 2024-03-25T19:25:44.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:50.029 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:03:39.601 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:50.081 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:03:39.654 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:50.125 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:03:39.698 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:50.137 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:03:39.709 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:50.141 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:03:39.674 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:50.145 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:03:39.679 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:50.149 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:03:39.721 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:50.183 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:03:39.717 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:50.184 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:03:39.756 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:51.153 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:51.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.249+0000 7fa2778c1640 10 monclient: get_monmap_and_config 2024-03-25T19:25:51.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.249+0000 7fa2778c1640 10 monclient: build_initial_monmap 2024-03-25T19:25:51.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.249+0000 7fa2778c1640 10 monclient: monmap: 2024-03-25T19:25:51.249 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:51.249 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:51.249 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:25:51.250371+0000 2024-03-25T19:25:51.249 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:25:51.250371+0000 2024-03-25T19:25:51.249 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:51.250 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:51.250 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:51.250 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:25:51.250 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:25:51.250 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.250+0000 7fa2778c1640 1 Processor -- start 2024-03-25T19:25:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.250+0000 7fa2778c1640 1 -- start start 2024-03-25T19:25:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.250+0000 7fa2778c1640 10 monclient: init 2024-03-25T19:25:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.251+0000 7fa2778c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.251+0000 7fa2778c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:25:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.251+0000 7fa2778c1640 10 monclient(hunting): picked mon.noname-b con 0x7fa2700b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.251+0000 7fa2778c1640 10 monclient(hunting): picked mon.noname-c con 0x7fa2700b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:51.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.251+0000 7fa2778c1640 10 monclient(hunting): picked mon.noname-a con 0x7fa27014be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:51.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.251+0000 7fa2778c1640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2700af940 con 0x7fa27014be90 2024-03-25T19:25:51.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.251+0000 7fa2778c1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2700b0a70 con 0x7fa2700b1730 2024-03-25T19:25:51.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.251+0000 7fa2778c1640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2700b0d50 con 0x7fa2700b4d60 2024-03-25T19:25:51.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.251+0000 7fa2778c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:51.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.251+0000 7fa2778c1640 10 monclient(hunting): authenticate will time out at 5788.611044s 2024-03-25T19:25:51.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.264+0000 7fa275e5e640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa27014be90 0x7fa27014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:52652/0 (socket says 172.21.15.123:52652) 2024-03-25T19:25:51.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.264+0000 7fa275e5e640 1 -- 172.21.15.123:0/1735158192 learned_addr learned my addr 172.21.15.123:0/1735158192 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:51.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.286+0000 7fa267fff640 1 -- 172.21.15.123:0/1735158192 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021771571 0 0) 0x7fa2700af940 con 0x7fa27014be90 2024-03-25T19:25:51.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.286+0000 7fa267fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:51.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.286+0000 7fa267fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:51.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.286+0000 7fa267fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:51.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.286+0000 7fa267fff640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:25:51.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.286+0000 7fa267fff640 1 -- 172.21.15.123:0/1735158192 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa25c003520 con 0x7fa27014be90 2024-03-25T19:25:51.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 1 -- 172.21.15.123:0/1735158192 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591350260 0 0) 0x7fa25c003520 con 0x7fa27014be90 2024-03-25T19:25:51.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 1 -- 172.21.15.123:0/1735158192 >> v1:172.21.15.123:6790/0 conn(0x7fa2700b4d60 legacy=0x7fa27013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:25:51.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 1 -- 172.21.15.123:0/1735158192 >> v1:172.21.15.151:6789/0 conn(0x7fa2700b1730 legacy=0x7fa2700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 10 monclient: _finish_hunting 0 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 1 monclient: found mon.noname-a 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 1 -- 172.21.15.123:0/1735158192 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2700b1030 con 0x7fa27014be90 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 10 monclient: _finish_auth 0 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 10 monclient: _check_auth_tickets 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 1 -- 172.21.15.123:0/1735158192 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa268001a10 con 0x7fa27014be90 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa2778c1640 5 monclient: authenticate success, global_id 4356 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa2778c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa267fff640 10 monclient: dump: 2024-03-25T19:25:51.300 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:51.301 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:51.301 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:51.301 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:51.301 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:51.301 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:51.301 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:51.301 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:51.301 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:51.301 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:51.301 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.299+0000 7fa2778c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:51.314 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.314+0000 7fa267fff640 1 -- 172.21.15.123:0/1735158192 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa268001be0 con 0x7fa27014be90 2024-03-25T19:25:51.314 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.314+0000 7fa267fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.314+0000 7fa267fff640 1 -- 172.21.15.123:0/1735158192 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa268002030 con 0x7fa27014be90 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.314+0000 7fa2778c1640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.314+0000 7fa2778c1640 4 set_mon_vals no callback set 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.314+0000 7fa267fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.314+0000 7fa267fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.314+0000 7fa267fff640 10 monclient: dump: 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:51.315 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:51.316 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:51.316 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:51.316 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:51.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.315+0000 7fa2778c1640 10 monclient: shutdown 2024-03-25T19:25:51.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.315+0000 7fa2778c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:51.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.315+0000 7fa2778c1640 1 -- 172.21.15.123:0/1735158192 >> v1:172.21.15.123:6789/0 conn(0x7fa27014be90 legacy=0x7fa27014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.315+0000 7fa2778c1640 1 -- 172.21.15.123:0/1735158192 shutdown_connections 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.315+0000 7fa2778c1640 1 -- 172.21.15.123:0/1735158192 >> 172.21.15.123:0/1735158192 conn(0x7fa2700617a0 msgr2=0x7fa270061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.315+0000 7fa2778c1640 1 -- 172.21.15.123:0/1735158192 shutdown_connections 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.315+0000 7fa2778c1640 1 -- 172.21.15.123:0/1735158192 wait complete. 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.316+0000 7fa2778c1640 10 monclient: build_initial_monmap 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.316+0000 7fa2778c1640 10 monclient: monmap: 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:51.318 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.316+0000 7fa2778c1640 1 librados: starting msgr at 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.316+0000 7fa2778c1640 1 librados: starting objecter 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 1 Processor -- start 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 1 -- start start 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 1 librados: setting wanted keys 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 1 librados: calling monclient init 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 10 monclient: init 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 10 monclient(hunting): picked mon.noname-b con 0x7fa2700b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 10 monclient(hunting): picked mon.noname-c con 0x7fa2700b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 10 monclient(hunting): picked mon.noname-a con 0x7fa27014be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:51.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2701403f0 con 0x7fa27014be90 2024-03-25T19:25:51.320 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa270140c60 con 0x7fa2700b1730 2024-03-25T19:25:51.320 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa270140df0 con 0x7fa2700b4d60 2024-03-25T19:25:51.320 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:51.320 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.317+0000 7fa2778c1640 10 monclient(hunting): authenticate will time out at 5788.677863s 2024-03-25T19:25:51.324 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.324+0000 7fa274e5c640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fa2700b4d60 0x7fa270136890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:52380/0 (socket says 172.21.15.123:52380) 2024-03-25T19:25:51.324 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.324+0000 7fa274e5c640 1 -- 172.21.15.123:0/2637820490 learned_addr learned my addr 172.21.15.123:0/2637820490 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:51.325 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.325+0000 7fa265ffb640 10 client.?.objecter ms_handle_connect 0x7fa2700b4d60 2024-03-25T19:25:51.325 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.325+0000 7fa265ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:51.332 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.332+0000 7fa265ffb640 10 client.?.objecter ms_handle_connect 0x7fa2700b1730 2024-03-25T19:25:51.332 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.332+0000 7fa265ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:51.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.335+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061268311 0 0) 0x7fa270140df0 con 0x7fa2700b4d60 2024-03-25T19:25:51.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.335+0000 7fa265ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:51.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.335+0000 7fa265ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:51.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.335+0000 7fa265ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:51.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.335+0000 7fa265ffb640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:25:51.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.335+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa254004eb0 con 0x7fa2700b4d60 2024-03-25T19:25:51.343 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.343+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 727817352 0 0) 0x7fa254004eb0 con 0x7fa2700b4d60 2024-03-25T19:25:51.343 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.343+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa270140df0 con 0x7fa2700b4d60 2024-03-25T19:25:51.343 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.343+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa260001a30 con 0x7fa2700b4d60 2024-03-25T19:25:51.343 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.343+0000 7fa265ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:51.343 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.343+0000 7fa265ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:51.343 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.343+0000 7fa265ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:51.344 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.343+0000 7fa265ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:51.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.345+0000 7fa265ffb640 10 client.?.objecter ms_handle_connect 0x7fa27014be90 2024-03-25T19:25:51.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.345+0000 7fa265ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:51.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.346+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897124496 0 0) 0x7fa270140c60 con 0x7fa2700b1730 2024-03-25T19:25:51.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.346+0000 7fa265ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:51.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.346+0000 7fa265ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:51.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.346+0000 7fa265ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:51.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.346+0000 7fa265ffb640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:25:51.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.346+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa254004eb0 con 0x7fa2700b1730 2024-03-25T19:25:51.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1210872144 0 0) 0x7fa270140df0 con 0x7fa2700b4d60 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 >> v1:172.21.15.151:6789/0 conn(0x7fa2700b1730 legacy=0x7fa270136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 >> v1:172.21.15.123:6789/0 conn(0x7fa27014be90 legacy=0x7fa2701e2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 1 monclient: found mon.c 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2701380f0 con 0x7fa2700b4d60 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 10 monclient: _finish_auth 0 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa265ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa2778c1640 5 monclient: authenticate success, global_id 4385 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa2778c1640 10 monclient: _renew_subs 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa2778c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa2778c1640 1 -- 172.21.15.123:0/2637820490 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa270138340 con 0x7fa2700b4d60 2024-03-25T19:25:51.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.354+0000 7fa2778c1640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:51.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.355+0000 7fa2778c1640 10 monclient: _renew_subs 2024-03-25T19:25:51.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.355+0000 7fa2778c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:51.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.355+0000 7fa2778c1640 1 -- 172.21.15.123:0/2637820490 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa270145520 con 0x7fa2700b4d60 2024-03-25T19:25:51.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.355+0000 7fa2778c1640 1 librados: init done 2024-03-25T19:25:51.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.356+0000 7fa2778c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:51.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.356+0000 7fa2778c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:51.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.356+0000 7fa2778c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:51.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.356+0000 7fa2778c1640 1 -- 172.21.15.123:0/2637820490 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2700627b0 con 0x7fa2700b4d60 2024-03-25T19:25:51.366 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.366+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa260001cf0 con 0x7fa2700b4d60 2024-03-25T19:25:51.366 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.366+0000 7fa265ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:51.367 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.366+0000 7fa2677fe640 4 set_mon_vals no callback set 2024-03-25T19:25:51.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.661+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa260003a60 con 0x7fa2700b4d60 2024-03-25T19:25:51.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.661+0000 7fa265ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:51.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.661+0000 7fa265ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:51.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.661+0000 7fa265ffb640 10 monclient: dump: 2024-03-25T19:25:51.661 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:51.661 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:51.661 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:51.661 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:51.661 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:51.661 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:51.662 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:51.662 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:51.662 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:51.662 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:51.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.661+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fa260003c80 con 0x7fa2700b4d60 2024-03-25T19:25:51.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.662+0000 7fa27565d640 10 monclient: get_auth_request con 0x7fa25406dd00 auth_method 0 2024-03-25T19:25:51.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.662+0000 7fa265ffb640 10 client.4385.objecter ms_handle_connect 0x7fa25406dd00 2024-03-25T19:25:51.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.668+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fa260054980 con 0x7fa2700b4d60 2024-03-25T19:25:51.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.668+0000 7fa265ffb640 10 client.4385.objecter ms_dispatch 0x7fa270000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:51.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.668+0000 7fa265ffb640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:51.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.668+0000 7fa265ffb640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:51.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.668+0000 7fa265ffb640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-25T19:25:51.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.668+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2600502c0 con 0x7fa2700b4d60 2024-03-25T19:25:51.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.668+0000 7fa265ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:51.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.668+0000 7fa265ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:51.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.920+0000 7fa2778c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:51.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.920+0000 7fa2778c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:51.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.920+0000 7fa2778c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:51.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.920+0000 7fa2778c1640 1 -- 172.21.15.123:0/2637820490 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa270138790 con 0x7fa2700b4d60 2024-03-25T19:25:51.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.945+0000 7fa265ffb640 1 -- 172.21.15.123:0/2637820490 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fa2600523c0 con 0x7fa2700b4d60 2024-03-25T19:25:51.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.945+0000 7fa265ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:51.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.945+0000 7fa265ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:51.945 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:25:51.945 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:51.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 10 librados: watch_flush enter 2024-03-25T19:25:51.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 10 librados: watch_flush exit 2024-03-25T19:25:51.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-25T19:25:51.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 10 client.4385.objecter successfully canceled tick 2024-03-25T19:25:51.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 1 -- 172.21.15.123:0/2637820490 >> v1:172.21.15.151:6800/727684239 conn(0x7fa25406dd00 legacy=0x7fa2540701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 10 monclient: shutdown 2024-03-25T19:25:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 1 -- 172.21.15.123:0/2637820490 >> v1:172.21.15.123:6790/0 conn(0x7fa2700b4d60 legacy=0x7fa270136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 1 -- 172.21.15.123:0/2637820490 shutdown_connections 2024-03-25T19:25:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 1 -- 172.21.15.123:0/2637820490 >> 172.21.15.123:0/2637820490 conn(0x7fa2700617a0 msgr2=0x7fa2700b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 1 -- 172.21.15.123:0/2637820490 shutdown_connections 2024-03-25T19:25:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 1 -- 172.21.15.123:0/2637820490 wait complete. 2024-03-25T19:25:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:51.948+0000 7fa2778c1640 1 librados: shutdown 2024-03-25T19:25:51.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:58.767 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:58.871 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.872+0000 7f7fb3d4d640 10 monclient: get_monmap_and_config 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.872+0000 7f7fb3d4d640 10 monclient: build_initial_monmap 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.872+0000 7f7fb3d4d640 10 monclient: monmap: 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:25:58.873049+0000 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:25:58.873049+0000 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:25:58.872 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:58.873 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.872+0000 7f7fb3d4d640 1 Processor -- start 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 1 -- start start 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 10 monclient: init 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f7fac0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f7fac0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f7fac14be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0af940 con 0x7f7fac14be90 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0b0a70 con 0x7f7fac0b1730 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0b0d50 con 0x7f7fac0b4d60 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:58.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.873+0000 7f7fb3d4d640 10 monclient(hunting): authenticate will time out at 5796.233716s 2024-03-25T19:25:58.886 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.887+0000 7f7fb22ea640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f7fac14be90 0x7f7fac14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:44460/0 (socket says 172.21.15.123:44460) 2024-03-25T19:25:58.886 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.887+0000 7f7fb22ea640 1 -- 172.21.15.123:0/1198000199 learned_addr learned my addr 172.21.15.123:0/1198000199 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:58.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.908+0000 7f7fb0ae7640 1 -- 172.21.15.123:0/1198000199 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726467145 0 0) 0x7f7fac0af940 con 0x7f7fac14be90 2024-03-25T19:25:58.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.908+0000 7f7fb0ae7640 10 monclient(hunting): _negotiate 2024-03-25T19:25:58.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.908+0000 7f7fb0ae7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:58.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.908+0000 7f7fb0ae7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:58.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.908+0000 7f7fb0ae7640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:25:58.909 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.908+0000 7f7fb0ae7640 1 -- 172.21.15.123:0/1198000199 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f98003520 con 0x7f7fac14be90 2024-03-25T19:25:58.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 1 -- 172.21.15.123:0/1198000199 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4026422958 0 0) 0x7f7f98003520 con 0x7f7fac14be90 2024-03-25T19:25:58.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 1 -- 172.21.15.123:0/1198000199 >> v1:172.21.15.123:6790/0 conn(0x7f7fac0b4d60 legacy=0x7f7fac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:25:58.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 1 -- 172.21.15.123:0/1198000199 >> v1:172.21.15.151:6789/0 conn(0x7f7fac0b1730 legacy=0x7f7fac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:58.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 10 monclient: _finish_hunting 0 2024-03-25T19:25:58.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 1 monclient: found mon.noname-a 2024-03-25T19:25:58.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 1 -- 172.21.15.123:0/1198000199 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac0b1030 con 0x7f7fac14be90 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 10 monclient: _finish_auth 0 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 10 monclient: _check_auth_tickets 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 1 -- 172.21.15.123:0/1198000199 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7fa4001a10 con 0x7f7fac14be90 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb3d4d640 5 monclient: authenticate success, global_id 4362 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb3d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb0ae7640 10 monclient: dump: 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:58.921 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:58.922 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:58.922 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:58.922 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:58.922 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:58.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.920+0000 7f7fb3d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:58.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.932+0000 7f7fb0ae7640 1 -- 172.21.15.123:0/1198000199 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fa4001be0 con 0x7f7fac14be90 2024-03-25T19:25:58.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.932+0000 7f7fb0ae7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:58.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.932+0000 7f7fb0ae7640 1 -- 172.21.15.123:0/1198000199 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7fa4002030 con 0x7f7fac14be90 2024-03-25T19:25:58.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.932+0000 7f7fb0ae7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:58.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.932+0000 7f7fb0ae7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.932+0000 7f7fb0ae7640 10 monclient: dump: 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.932+0000 7f7fb3d4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:58.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.932+0000 7f7fb3d4d640 4 set_mon_vals no callback set 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.933+0000 7f7fb3d4d640 10 monclient: shutdown 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.933+0000 7f7fb3d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.933+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1198000199 >> v1:172.21.15.123:6789/0 conn(0x7f7fac14be90 legacy=0x7f7fac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.933+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1198000199 shutdown_connections 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.933+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1198000199 >> 172.21.15.123:0/1198000199 conn(0x7f7fac0617a0 msgr2=0x7f7fac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.933+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1198000199 shutdown_connections 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.933+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1198000199 wait complete. 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.934+0000 7f7fb3d4d640 10 monclient: build_initial_monmap 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.934+0000 7f7fb3d4d640 10 monclient: monmap: 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:25:58.936 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.934+0000 7f7fb3d4d640 1 librados: starting msgr at 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.934+0000 7f7fb3d4d640 1 librados: starting objecter 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.935+0000 7f7fb3d4d640 1 Processor -- start 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.935+0000 7f7fb3d4d640 1 -- start start 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.935+0000 7f7fb3d4d640 1 librados: setting wanted keys 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.935+0000 7f7fb3d4d640 1 librados: calling monclient init 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.935+0000 7f7fb3d4d640 10 monclient: init 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.935+0000 7f7fb3d4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:58.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.935+0000 7f7fb3d4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:25:58.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.935+0000 7f7fb3d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f7fac0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:25:58.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.935+0000 7f7fb3d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f7fac0b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:25:58.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.935+0000 7f7fb3d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f7fac14be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:25:58.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.936+0000 7f7fb3d4d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac1daf60 con 0x7f7fac0b4d60 2024-03-25T19:25:58.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.936+0000 7f7fb3d4d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac1422d0 con 0x7f7fac0b1730 2024-03-25T19:25:58.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.936+0000 7f7fb3d4d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac140a90 con 0x7f7fac14be90 2024-03-25T19:25:58.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.936+0000 7f7fb3d4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:58.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.936+0000 7f7fb3d4d640 10 monclient(hunting): authenticate will time out at 5796.296006s 2024-03-25T19:25:58.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.938+0000 7f7fb1ae9640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f7fac0b1730 0x7f7fac140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:46538/0 (socket says 172.21.15.123:46538) 2024-03-25T19:25:58.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.938+0000 7f7fb1ae9640 1 -- 172.21.15.123:0/1191125491 learned_addr learned my addr 172.21.15.123:0/1191125491 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:25:58.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.938+0000 7f7fa27fc640 10 client.?.objecter ms_handle_connect 0x7f7fac0b1730 2024-03-25T19:25:58.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.938+0000 7f7fa27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:58.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.950+0000 7f7fa27fc640 10 client.?.objecter ms_handle_connect 0x7f7fac14be90 2024-03-25T19:25:58.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.950+0000 7f7fa27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:58.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.953+0000 7f7fa27fc640 10 client.?.objecter ms_handle_connect 0x7f7fac0b4d60 2024-03-25T19:25:58.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:58.953+0000 7f7fa27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:59.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.019+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311613665 0 0) 0x7f7fac1422d0 con 0x7f7fac0b1730 2024-03-25T19:25:59.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.019+0000 7f7fa27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:59.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.019+0000 7f7fa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:59.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.019+0000 7f7fa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:59.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.019+0000 7f7fa27fc640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:25:59.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.019+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f90004eb0 con 0x7f7fac0b1730 2024-03-25T19:25:59.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.021+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393942781 0 0) 0x7f7fac140a90 con 0x7f7fac14be90 2024-03-25T19:25:59.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.021+0000 7f7fa27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:59.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.021+0000 7f7fa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:59.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.021+0000 7f7fa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:59.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.021+0000 7f7fa27fc640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:25:59.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.021+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fac1422d0 con 0x7f7fac14be90 2024-03-25T19:25:59.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.022+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788812900 0 0) 0x7f7fac1daf60 con 0x7f7fac0b4d60 2024-03-25T19:25:59.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.022+0000 7f7fa27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:59.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.022+0000 7f7fa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:59.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.022+0000 7f7fa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:59.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.022+0000 7f7fa27fc640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:25:59.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.022+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fac140a90 con 0x7f7fac0b4d60 2024-03-25T19:25:59.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.032+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3938444598 0 0) 0x7f7f90004eb0 con 0x7f7fac0b1730 2024-03-25T19:25:59.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.032+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fac1daf60 con 0x7f7fac0b1730 2024-03-25T19:25:59.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.032+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7fa8001960 con 0x7f7fac0b1730 2024-03-25T19:25:59.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.032+0000 7f7fa27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:59.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.032+0000 7f7fa27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:59.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.032+0000 7f7fa27fc640 10 monclient(hunting): dump: 2024-03-25T19:25:59.032 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:59.032 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:59.033 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:59.033 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:59.033 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:59.033 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:59.033 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:59.033 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:59.033 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:59.033 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:59.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.032+0000 7f7fa27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:59.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.039+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2743313178 0 0) 0x7f7fac1422d0 con 0x7f7fac14be90 2024-03-25T19:25:59.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.039+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f90004eb0 con 0x7f7fac14be90 2024-03-25T19:25:59.039 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.039+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7fa4001ef0 con 0x7f7fac14be90 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.039+0000 7f7fa27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.039+0000 7f7fa27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.039+0000 7f7fa27fc640 10 monclient(hunting): dump: 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:59.040 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:59.041 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.040+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1259239136 0 0) 0x7f7fac140a90 con 0x7f7fac0b4d60 2024-03-25T19:25:59.041 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.040+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fac1422d0 con 0x7f7fac0b4d60 2024-03-25T19:25:59.041 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.040+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7f9c001830 con 0x7f7fac0b4d60 2024-03-25T19:25:59.041 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.040+0000 7f7fa27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:59.041 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.040+0000 7f7fa27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:59.041 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.040+0000 7f7fa27fc640 10 monclient(hunting): dump: 2024-03-25T19:25:59.041 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:59.041 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:59.042 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:59.042 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:59.042 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:59.042 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:59.042 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:59.042 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:59.042 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:59.042 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:59.051 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 370171323 0 0) 0x7f7f90004eb0 con 0x7f7fac14be90 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 >> v1:172.21.15.151:6789/0 conn(0x7f7fac0b1730 legacy=0x7f7fac140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 >> v1:172.21.15.123:6789/0 conn(0x7f7fac0b4d60 legacy=0x7f7fac1db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 1 monclient: found mon.c 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac1d79e0 con 0x7f7fac14be90 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 10 monclient: _finish_auth 0 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fb3d4d640 5 monclient: authenticate success, global_id 4388 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fb3d4d640 10 monclient: _renew_subs 2024-03-25T19:25:59.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fb3d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:59.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.051+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac1dbb00 con 0x7f7fac14be90 2024-03-25T19:25:59.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.052+0000 7f7fb3d4d640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:59.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.052+0000 7f7fb3d4d640 10 monclient: _renew_subs 2024-03-25T19:25:59.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.052+0000 7f7fb3d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:59.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.052+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac1d87a0 con 0x7f7fac14be90 2024-03-25T19:25:59.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.052+0000 7f7fb3d4d640 1 librados: init done 2024-03-25T19:25:59.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.052+0000 7f7fb3d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:59.054 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.052+0000 7f7fb3d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:59.054 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.052+0000 7f7fb3d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:59.054 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.052+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac140c20 con 0x7f7fac14be90 2024-03-25T19:25:59.359 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.359+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fa4002050 con 0x7f7fac14be90 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.359+0000 7f7fa27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.359+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7fa4004fb0 con 0x7f7fac14be90 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.359+0000 7f7fa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.359+0000 7f7fa27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.359+0000 7f7fa27fc640 10 monclient: dump: 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:59.360 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:25:59.361 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:25:59.361 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:25:59.361 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:25:59.361 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:25:59.361 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.359+0000 7f7fa3fff640 4 set_mon_vals no callback set 2024-03-25T19:25:59.361 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.360+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f7fa401af80 con 0x7f7fac14be90 2024-03-25T19:25:59.361 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.361+0000 7f7fb1ae9640 10 monclient: get_auth_request con 0x7f7f9006e080 auth_method 0 2024-03-25T19:25:59.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.361+0000 7f7fa27fc640 10 client.4388.objecter ms_handle_connect 0x7f7f9006e080 2024-03-25T19:25:59.366 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.366+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f7fa4054ac0 con 0x7f7fac14be90 2024-03-25T19:25:59.366 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.366+0000 7f7fa27fc640 10 client.4388.objecter ms_dispatch 0x7f7fac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:59.366 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.366+0000 7f7fa27fc640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:59.366 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.366+0000 7f7fa27fc640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:59.366 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.366+0000 7f7fa27fc640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T19:25:59.367 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.366+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fa4050400 con 0x7f7fac14be90 2024-03-25T19:25:59.367 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.366+0000 7f7fa27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:59.367 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.366+0000 7f7fa27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:59.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.622+0000 7f7fb3d4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:59.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.622+0000 7f7fb3d4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:59.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.622+0000 7f7fb3d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:25:59.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.622+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1191125491 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fac1dbe50 con 0x7f7fac14be90 2024-03-25T19:25:59.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.659+0000 7f7fa27fc640 1 -- 172.21.15.123:0/1191125491 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f7fa4052500 con 0x7f7fac14be90 2024-03-25T19:25:59.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.659+0000 7f7fa27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:59.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.659+0000 7f7fa27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:59.659 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:25:59.659 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:59.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 10 librados: watch_flush enter 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 10 librados: watch_flush exit 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 10 client.4388.objecter successfully canceled tick 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1191125491 >> v1:172.21.15.151:6800/727684239 conn(0x7f7f9006e080 legacy=0x7f7f90070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 10 monclient: shutdown 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1191125491 >> v1:172.21.15.123:6790/0 conn(0x7f7fac14be90 legacy=0x7f7fac1d7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1191125491 shutdown_connections 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1191125491 >> 172.21.15.123:0/1191125491 conn(0x7f7fac0617a0 msgr2=0x7f7fac145130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1191125491 shutdown_connections 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 1 -- 172.21.15.123:0/1191125491 wait complete. 2024-03-25T19:25:59.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:25:59.661+0000 7f7fb3d4d640 1 librados: shutdown 2024-03-25T19:25:59.673 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:06.480 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:06.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.595+0000 7f32f9158640 10 monclient: get_monmap_and_config 2024-03-25T19:26:06.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.595+0000 7f32f9158640 10 monclient: build_initial_monmap 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.595+0000 7f32f9158640 10 monclient: monmap: 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:26:06.596009+0000 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:26:06.596009+0000 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:26:06.595 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:06.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.595+0000 7f32f9158640 1 Processor -- start 2024-03-25T19:26:06.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.595+0000 7f32f9158640 1 -- start start 2024-03-25T19:26:06.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.595+0000 7f32f9158640 10 monclient: init 2024-03-25T19:26:06.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.596+0000 7f32f9158640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:06.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.596+0000 7f32f9158640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:26:06.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.596+0000 7f32f9158640 10 monclient(hunting): picked mon.noname-a con 0x7f32f40b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:06.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.596+0000 7f32f9158640 10 monclient(hunting): picked mon.noname-b con 0x7f32f40b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:06.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.596+0000 7f32f9158640 10 monclient(hunting): picked mon.noname-c con 0x7f32f414be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:06.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.596+0000 7f32f9158640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f40af940 con 0x7f32f40b4d60 2024-03-25T19:26:06.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.596+0000 7f32f9158640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f40b0a70 con 0x7f32f414be90 2024-03-25T19:26:06.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.596+0000 7f32f9158640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f40b0d50 con 0x7f32f40b1730 2024-03-25T19:26:06.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.596+0000 7f32f9158640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:06.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.596+0000 7f32f9158640 10 monclient(hunting): authenticate will time out at 5803.956251s 2024-03-25T19:26:06.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.615+0000 7f32f359e640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f32f414be90 0x7f32f414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:51644/0 (socket says 172.21.15.123:51644) 2024-03-25T19:26:06.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.615+0000 7f32f359e640 1 -- 172.21.15.123:0/3834198630 learned_addr learned my addr 172.21.15.123:0/3834198630 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:06.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.627+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174039595 0 0) 0x7f32f40b0d50 con 0x7f32f40b1730 2024-03-25T19:26:06.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.627+0000 7f32f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:26:06.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.627+0000 7f32f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:06.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.627+0000 7f32f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:06.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.627+0000 7f32f1d9b640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:26:06.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.627+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32dc003520 con 0x7f32f40b1730 2024-03-25T19:26:06.635 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.636+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277758159 0 0) 0x7f32f40af940 con 0x7f32f40b4d60 2024-03-25T19:26:06.636 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.636+0000 7f32f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:26:06.636 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.636+0000 7f32f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:06.636 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.636+0000 7f32f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:06.636 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.636+0000 7f32f1d9b640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:26:06.636 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.636+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f40b0d50 con 0x7f32f40b4d60 2024-03-25T19:26:06.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3195364105 0 0) 0x7f32dc003520 con 0x7f32f40b1730 2024-03-25T19:26:06.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 >> v1:172.21.15.151:6789/0 conn(0x7f32f414be90 legacy=0x7f32f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 >> v1:172.21.15.123:6789/0 conn(0x7f32f40b4d60 legacy=0x7f32f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:26:06.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 1 monclient: found mon.noname-b 2024-03-25T19:26:06.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:06.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:26:06.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f40b1030 con 0x7f32f40b1730 2024-03-25T19:26:06.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 10 monclient: _finish_auth 0 2024-03-25T19:26:06.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f32e8001690 con 0x7f32f40b1730 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f9158640 5 monclient: authenticate success, global_id 4391 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f9158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f1d9b640 10 monclient: dump: 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:06.646 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:06.647 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:06.647 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:06.647 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.645+0000 7f32f9158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:06.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.652+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32e80017f0 con 0x7f32f40b1730 2024-03-25T19:26:06.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.652+0000 7f32f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:06.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.652+0000 7f32f1d9b640 1 -- 172.21.15.123:0/3834198630 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f32e8001c00 con 0x7f32f40b1730 2024-03-25T19:26:06.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.652+0000 7f32f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:06.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.652+0000 7f32f1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.652+0000 7f32f1d9b640 10 monclient: dump: 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:06.653 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.652+0000 7f32f9158640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:06.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.652+0000 7f32f9158640 4 set_mon_vals no callback set 2024-03-25T19:26:06.656 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.653+0000 7f32f9158640 10 monclient: shutdown 2024-03-25T19:26:06.656 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.653+0000 7f32f9158640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:06.656 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.653+0000 7f32f9158640 1 -- 172.21.15.123:0/3834198630 >> v1:172.21.15.123:6790/0 conn(0x7f32f40b1730 legacy=0x7f32f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.656 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.653+0000 7f32f9158640 1 -- 172.21.15.123:0/3834198630 shutdown_connections 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.653+0000 7f32f9158640 1 -- 172.21.15.123:0/3834198630 >> 172.21.15.123:0/3834198630 conn(0x7f32f40617a0 msgr2=0x7f32f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.653+0000 7f32f9158640 1 -- 172.21.15.123:0/3834198630 shutdown_connections 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.653+0000 7f32f9158640 1 -- 172.21.15.123:0/3834198630 wait complete. 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.654+0000 7f32f9158640 10 monclient: build_initial_monmap 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.654+0000 7f32f9158640 10 monclient: monmap: 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:26:06.657 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.654+0000 7f32f9158640 1 librados: starting msgr at 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.654+0000 7f32f9158640 1 librados: starting objecter 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.654+0000 7f32f9158640 1 Processor -- start 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.655+0000 7f32f9158640 1 -- start start 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.655+0000 7f32f9158640 1 librados: setting wanted keys 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.655+0000 7f32f9158640 1 librados: calling monclient init 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.655+0000 7f32f9158640 10 monclient: init 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.655+0000 7f32f9158640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.655+0000 7f32f9158640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.655+0000 7f32f9158640 10 monclient(hunting): picked mon.noname-b con 0x7f32f40b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.655+0000 7f32f9158640 10 monclient(hunting): picked mon.noname-c con 0x7f32f40b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.656+0000 7f32f9158640 10 monclient(hunting): picked mon.noname-a con 0x7f32f414be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:06.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.656+0000 7f32f9158640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f4137860 con 0x7f32f414be90 2024-03-25T19:26:06.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.656+0000 7f32f9158640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f4140800 con 0x7f32f40b1730 2024-03-25T19:26:06.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.656+0000 7f32f9158640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f4137cf0 con 0x7f32f40b4d60 2024-03-25T19:26:06.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.656+0000 7f32f9158640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:06.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.656+0000 7f32f9158640 10 monclient(hunting): authenticate will time out at 5804.015948s 2024-03-25T19:26:06.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.656+0000 7f32f2d9d640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f32f40b1730 0x7f32f4136880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:51678/0 (socket says 172.21.15.123:51678) 2024-03-25T19:26:06.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.656+0000 7f32f2d9d640 1 -- 172.21.15.123:0/3150254575 learned_addr learned my addr 172.21.15.123:0/3150254575 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:06.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.656+0000 7f32db7fe640 10 client.?.objecter ms_handle_connect 0x7f32f40b1730 2024-03-25T19:26:06.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.656+0000 7f32db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:06.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.657+0000 7f32db7fe640 10 client.?.objecter ms_handle_connect 0x7f32f40b4d60 2024-03-25T19:26:06.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.657+0000 7f32db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:06.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.718+0000 7f32db7fe640 10 client.?.objecter ms_handle_connect 0x7f32f414be90 2024-03-25T19:26:06.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.718+0000 7f32db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:06.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.730+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743657358 0 0) 0x7f32f4137860 con 0x7f32f414be90 2024-03-25T19:26:06.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.730+0000 7f32db7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:06.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.730+0000 7f32db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:06.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.730+0000 7f32db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:06.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.730+0000 7f32db7fe640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:26:06.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.730+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e0004eb0 con 0x7f32f414be90 2024-03-25T19:26:06.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.741+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 119208374 0 0) 0x7f32e0004eb0 con 0x7f32f414be90 2024-03-25T19:26:06.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.741+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32f4137860 con 0x7f32f414be90 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.741+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f32ec001b00 con 0x7f32f414be90 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.741+0000 7f32db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.741+0000 7f32db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.741+0000 7f32db7fe640 10 monclient(hunting): dump: 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:06.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.741+0000 7f32db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:06.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.771+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894554098 0 0) 0x7f32f4140800 con 0x7f32f40b1730 2024-03-25T19:26:06.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.771+0000 7f32db7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:06.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.771+0000 7f32db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:06.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.771+0000 7f32db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:06.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.771+0000 7f32db7fe640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:26:06.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.771+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e0004eb0 con 0x7f32f40b1730 2024-03-25T19:26:06.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4068069681 0 0) 0x7f32f4137860 con 0x7f32f414be90 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 >> v1:172.21.15.123:6790/0 conn(0x7f32f40b4d60 legacy=0x7f32f4143ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 >> v1:172.21.15.151:6789/0 conn(0x7f32f40b1730 legacy=0x7f32f4136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 1 monclient: found mon.a 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f4138920 con 0x7f32f414be90 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 10 monclient: _finish_auth 0 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32f9158640 5 monclient: authenticate success, global_id 4371 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32f9158640 10 monclient: _renew_subs 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32f9158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.776+0000 7f32f9158640 1 -- 172.21.15.123:0/3150254575 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f41445b0 con 0x7f32f414be90 2024-03-25T19:26:06.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.777+0000 7f32f9158640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:06.778 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.777+0000 7f32f9158640 10 monclient: _renew_subs 2024-03-25T19:26:06.778 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.777+0000 7f32f9158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:06.778 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.777+0000 7f32f9158640 1 -- 172.21.15.123:0/3150254575 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f4137240 con 0x7f32f414be90 2024-03-25T19:26:06.778 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.777+0000 7f32f9158640 1 librados: init done 2024-03-25T19:26:06.778 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.777+0000 7f32f9158640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:06.778 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.777+0000 7f32f9158640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:06.778 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.777+0000 7f32f9158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:06.779 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:06.777+0000 7f32f9158640 1 -- 172.21.15.123:0/3150254575 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f4140670 con 0x7f32f414be90 2024-03-25T19:26:07.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.069+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32ec001fb0 con 0x7f32f414be90 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.069+0000 7f32db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.069+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f32ec004d00 con 0x7f32f414be90 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.069+0000 7f32db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.069+0000 7f32db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.069+0000 7f32db7fe640 10 monclient: dump: 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:07.070 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:07.071 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:07.071 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:07.071 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:07.071 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:07.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.069+0000 7f32f159a640 4 set_mon_vals no callback set 2024-03-25T19:26:07.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.070+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f32ec004f20 con 0x7f32f414be90 2024-03-25T19:26:07.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.071+0000 7f32f2d9d640 10 monclient: get_auth_request con 0x7f32e006de20 auth_method 0 2024-03-25T19:26:07.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.071+0000 7f32db7fe640 10 client.4371.objecter ms_handle_connect 0x7f32e006de20 2024-03-25T19:26:07.077 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.077+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f32ec0548f0 con 0x7f32f414be90 2024-03-25T19:26:07.077 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.077+0000 7f32db7fe640 10 client.4371.objecter ms_dispatch 0x7f32f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:07.077 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.077+0000 7f32db7fe640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:07.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.077+0000 7f32db7fe640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:07.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.077+0000 7f32db7fe640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T19:26:07.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.077+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32ec050230 con 0x7f32f414be90 2024-03-25T19:26:07.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.077+0000 7f32db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:07.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.077+0000 7f32db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:07.334 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.334+0000 7f32f9158640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:07.334 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.334+0000 7f32f9158640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:07.334 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.334+0000 7f32f9158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:07.334 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.334+0000 7f32f9158640 1 -- 172.21.15.123:0/3150254575 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32f4137a10 con 0x7f32f414be90 2024-03-25T19:26:07.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.360+0000 7f32db7fe640 1 -- 172.21.15.123:0/3150254575 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f32f4137a10 con 0x7f32f414be90 2024-03-25T19:26:07.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.360+0000 7f32db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:07.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.360+0000 7f32db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:07.360 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:26:07.360 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:07.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.362+0000 7f32f9158640 10 librados: watch_flush enter 2024-03-25T19:26:07.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.362+0000 7f32f9158640 10 librados: watch_flush exit 2024-03-25T19:26:07.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.362+0000 7f32f9158640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T19:26:07.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.362+0000 7f32f9158640 10 client.4371.objecter successfully canceled tick 2024-03-25T19:26:07.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.362+0000 7f32f9158640 1 -- 172.21.15.123:0/3150254575 >> v1:172.21.15.151:6800/727684239 conn(0x7f32e006de20 legacy=0x7f32e00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:07.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.362+0000 7f32f9158640 10 monclient: shutdown 2024-03-25T19:26:07.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.362+0000 7f32f9158640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:07.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.362+0000 7f32f9158640 1 -- 172.21.15.123:0/3150254575 >> v1:172.21.15.123:6789/0 conn(0x7f32f414be90 legacy=0x7f32f41e28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:07.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.363+0000 7f32f9158640 1 -- 172.21.15.123:0/3150254575 shutdown_connections 2024-03-25T19:26:07.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.363+0000 7f32f9158640 1 -- 172.21.15.123:0/3150254575 >> 172.21.15.123:0/3150254575 conn(0x7f32f40617a0 msgr2=0x7f32f413c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:07.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.363+0000 7f32f9158640 1 -- 172.21.15.123:0/3150254575 shutdown_connections 2024-03-25T19:26:07.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.363+0000 7f32f9158640 1 -- 172.21.15.123:0/3150254575 wait complete. 2024-03-25T19:26:07.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:07.363+0000 7f32f9158640 1 librados: shutdown 2024-03-25T19:26:07.375 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:10.187 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:03:59.759 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:10.194 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:03:59.766 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:10.232 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:03:59.804 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:10.247 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:03:59.781 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:10.255 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:03:59.789 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:10.280 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:03:59.852 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:10.290 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:03:59.862 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:10.296 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:03:59.830 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:10.316 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:03:59.888 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:14.181 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.294+0000 7f15794a8640 10 monclient: get_monmap_and_config 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.294+0000 7f15794a8640 10 monclient: build_initial_monmap 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.294+0000 7f15794a8640 10 monclient: monmap: 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:26:14.295563+0000 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:26:14.295563+0000 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:14.296 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.295+0000 7f15794a8640 1 Processor -- start 2024-03-25T19:26:14.296 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.295+0000 7f15794a8640 1 -- start start 2024-03-25T19:26:14.296 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.295+0000 7f15794a8640 10 monclient: init 2024-03-25T19:26:14.297 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.296+0000 7f15794a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:14.297 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.296+0000 7f15794a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:14.297 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.296+0000 7f15794a8640 10 monclient(hunting): picked mon.noname-b con 0x7f15740a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:14.297 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.296+0000 7f15794a8640 10 monclient(hunting): picked mon.noname-c con 0x7f157413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:14.297 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.296+0000 7f15794a8640 10 monclient(hunting): picked mon.noname-a con 0x7f15741430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:14.297 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.296+0000 7f15794a8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1574142520 con 0x7f15741430c0 2024-03-25T19:26:14.297 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.296+0000 7f15794a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15740b28a0 con 0x7f157413f710 2024-03-25T19:26:14.297 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.296+0000 7f15794a8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15740b2a30 con 0x7f15740a98f0 2024-03-25T19:26:14.297 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.296+0000 7f15794a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:14.297 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.296+0000 7f15794a8640 10 monclient(hunting): authenticate will time out at 5811.656486s 2024-03-25T19:26:14.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.312+0000 7f1572ffd640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f15740a98f0 0x7f15740a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:35580/0 (socket says 172.21.15.123:35580) 2024-03-25T19:26:14.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.312+0000 7f1572ffd640 1 -- 172.21.15.123:0/617399307 learned_addr learned my addr 172.21.15.123:0/617399307 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:14.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.329+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032298174 0 0) 0x7f15740b2a30 con 0x7f15740a98f0 2024-03-25T19:26:14.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.329+0000 7f1571ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:14.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.329+0000 7f1571ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:14.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.329+0000 7f1571ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:14.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.329+0000 7f1571ffb640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:26:14.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.329+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f155c003520 con 0x7f15740a98f0 2024-03-25T19:26:14.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.330+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831028960 0 0) 0x7f15740b28a0 con 0x7f157413f710 2024-03-25T19:26:14.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.330+0000 7f1571ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:14.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.330+0000 7f1571ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:14.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.330+0000 7f1571ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:14.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.330+0000 7f1571ffb640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:26:14.331 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.330+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15740b2a30 con 0x7f157413f710 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 703099832 0 0) 0x7f15740b2a30 con 0x7f157413f710 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 >> v1:172.21.15.123:6790/0 conn(0x7f15740a98f0 legacy=0x7f15740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 >> v1:172.21.15.123:6789/0 conn(0x7f15741430c0 legacy=0x7f157414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 1 monclient: found mon.noname-c 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15740b44c0 con 0x7f157413f710 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 10 monclient: _finish_auth 0 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1568001670 con 0x7f157413f710 2024-03-25T19:26:14.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f1571ffb640 10 monclient: dump: 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.334+0000 7f15794a8640 5 monclient: authenticate success, global_id 4387 2024-03-25T19:26:14.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.335+0000 7f15794a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:14.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.351+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15680017d0 con 0x7f157413f710 2024-03-25T19:26:14.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.351+0000 7f1571ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:14.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.351+0000 7f1571ffb640 1 -- 172.21.15.123:0/617399307 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1568001c20 con 0x7f157413f710 2024-03-25T19:26:14.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.351+0000 7f15794a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:14.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.351+0000 7f15794a8640 4 set_mon_vals no callback set 2024-03-25T19:26:14.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.351+0000 7f1571ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:14.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.351+0000 7f1571ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:14.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.351+0000 7f1571ffb640 10 monclient: dump: 2024-03-25T19:26:14.351 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:14.352 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:14.352 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.352 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.352 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:14.352 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:14.352 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:14.352 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:14.352 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:14.352 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:14.353 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.352+0000 7f15794a8640 10 monclient: shutdown 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.352+0000 7f15794a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.352+0000 7f15794a8640 1 -- 172.21.15.123:0/617399307 >> v1:172.21.15.151:6789/0 conn(0x7f157413f710 legacy=0x7f15741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.352+0000 7f15794a8640 1 -- 172.21.15.123:0/617399307 shutdown_connections 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.352+0000 7f15794a8640 1 -- 172.21.15.123:0/617399307 >> 172.21.15.123:0/617399307 conn(0x7f15740b7ea0 msgr2=0x7f15740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.352+0000 7f15794a8640 1 -- 172.21.15.123:0/617399307 shutdown_connections 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.352+0000 7f15794a8640 1 -- 172.21.15.123:0/617399307 wait complete. 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.353+0000 7f15794a8640 10 monclient: build_initial_monmap 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.353+0000 7f15794a8640 10 monclient: monmap: 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:26:14.354 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.353+0000 7f15794a8640 1 librados: starting msgr at 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.353+0000 7f15794a8640 1 librados: starting objecter 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 1 Processor -- start 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 1 -- start start 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 1 librados: setting wanted keys 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 1 librados: calling monclient init 2024-03-25T19:26:14.355 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 10 monclient: init 2024-03-25T19:26:14.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:14.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:26:14.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 10 monclient(hunting): picked mon.noname-b con 0x7f15740a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:14.356 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 10 monclient(hunting): picked mon.noname-a con 0x7f157413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:14.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 10 monclient(hunting): picked mon.noname-c con 0x7f15741430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:14.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1574139730 con 0x7f157413f710 2024-03-25T19:26:14.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1574138e00 con 0x7f15740a98f0 2024-03-25T19:26:14.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15741399a0 con 0x7f15741430c0 2024-03-25T19:26:14.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:14.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.354+0000 7f15794a8640 10 monclient(hunting): authenticate will time out at 5811.714619s 2024-03-25T19:26:14.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.358+0000 7f15737fe640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f15741430c0 0x7f1574133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:35646/0 (socket says 172.21.15.123:35646) 2024-03-25T19:26:14.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.358+0000 7f15737fe640 1 -- 172.21.15.123:0/1621550524 learned_addr learned my addr 172.21.15.123:0/1621550524 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:14.359 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.360+0000 7f155b7fe640 10 client.?.objecter ms_handle_connect 0x7f15741430c0 2024-03-25T19:26:14.359 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.360+0000 7f155b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:14.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.363+0000 7f155b7fe640 10 client.?.objecter ms_handle_connect 0x7f157413f710 2024-03-25T19:26:14.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.363+0000 7f155b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:14.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.369+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771702396 0 0) 0x7f1574139730 con 0x7f157413f710 2024-03-25T19:26:14.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.369+0000 7f155b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:14.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.369+0000 7f155b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:14.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.369+0000 7f155b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:14.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.369+0000 7f155b7fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:26:14.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.369+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1560004eb0 con 0x7f157413f710 2024-03-25T19:26:14.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.372+0000 7f155b7fe640 10 client.?.objecter ms_handle_connect 0x7f15740a98f0 2024-03-25T19:26:14.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.372+0000 7f155b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:14.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.380+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157826083 0 0) 0x7f15741399a0 con 0x7f15741430c0 2024-03-25T19:26:14.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.380+0000 7f155b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:14.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.380+0000 7f155b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:14.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.380+0000 7f155b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:14.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.380+0000 7f155b7fe640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:26:14.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.380+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1574139730 con 0x7f15741430c0 2024-03-25T19:26:14.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.384+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092045465 0 0) 0x7f1574138e00 con 0x7f15740a98f0 2024-03-25T19:26:14.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.384+0000 7f155b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:14.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.384+0000 7f155b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:14.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.384+0000 7f155b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:14.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.384+0000 7f155b7fe640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:26:14.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.384+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15741399a0 con 0x7f15740a98f0 2024-03-25T19:26:14.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.386+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3158323280 0 0) 0x7f1560004eb0 con 0x7f157413f710 2024-03-25T19:26:14.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.386+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1574138e00 con 0x7f157413f710 2024-03-25T19:26:14.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.386+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1568001ba0 con 0x7f157413f710 2024-03-25T19:26:14.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.386+0000 7f155b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.386+0000 7f155b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.386+0000 7f155b7fe640 10 monclient(hunting): dump: 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:14.387 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.386+0000 7f155b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:14.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.398+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4047015170 0 0) 0x7f1574139730 con 0x7f15741430c0 2024-03-25T19:26:14.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.398+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1560004eb0 con 0x7f15741430c0 2024-03-25T19:26:14.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.398+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f156c001bd0 con 0x7f15741430c0 2024-03-25T19:26:14.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.398+0000 7f155b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.398+0000 7f155b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.398+0000 7f155b7fe640 10 monclient(hunting): dump: 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:14.399 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:14.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.402+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3778960945 0 0) 0x7f15741399a0 con 0x7f15740a98f0 2024-03-25T19:26:14.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.402+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1574139730 con 0x7f15740a98f0 2024-03-25T19:26:14.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.402+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1564001950 con 0x7f15740a98f0 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.402+0000 7f155b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.402+0000 7f155b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.402+0000 7f155b7fe640 10 monclient(hunting): dump: 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:14.403 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:14.404 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2661152418 0 0) 0x7f1574138e00 con 0x7f157413f710 2024-03-25T19:26:14.404 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 >> v1:172.21.15.123:6790/0 conn(0x7f15741430c0 legacy=0x7f1574133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:14.404 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 >> v1:172.21.15.151:6789/0 conn(0x7f15740a98f0 legacy=0x7f1574135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 1 monclient: found mon.a 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1574137cc0 con 0x7f157413f710 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f155b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f15794a8640 5 monclient: authenticate success, global_id 4377 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f15794a8640 10 monclient: _renew_subs 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f15794a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f15794a8640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1574137900 con 0x7f157413f710 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f15794a8640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f15794a8640 10 monclient: _renew_subs 2024-03-25T19:26:14.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f15794a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:14.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f15794a8640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1574138890 con 0x7f157413f710 2024-03-25T19:26:14.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.404+0000 7f15794a8640 1 librados: init done 2024-03-25T19:26:14.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.405+0000 7f15794a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:14.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.405+0000 7f15794a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:14.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.405+0000 7f15794a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:14.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.405+0000 7f15794a8640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15740627b0 con 0x7f157413f710 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.717+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15680038a0 con 0x7f157413f710 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.717+0000 7f155b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.717+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1568003bb0 con 0x7f157413f710 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.717+0000 7f155b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.717+0000 7f155b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.717+0000 7f155b7fe640 10 monclient: dump: 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:14.717 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:14.718 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:14.718 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:14.718 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:14.718 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:14.718 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:14.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.717+0000 7f15717fa640 4 set_mon_vals no callback set 2024-03-25T19:26:14.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.718+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f1568019b80 con 0x7f157413f710 2024-03-25T19:26:14.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.718+0000 7f1572ffd640 10 monclient: get_auth_request con 0x7f156006e0b0 auth_method 0 2024-03-25T19:26:14.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.719+0000 7f155b7fe640 10 client.4377.objecter ms_handle_connect 0x7f156006e0b0 2024-03-25T19:26:14.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.724+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f15680536d0 con 0x7f157413f710 2024-03-25T19:26:14.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.724+0000 7f155b7fe640 10 client.4377.objecter ms_dispatch 0x7f1574000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:14.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.724+0000 7f155b7fe640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:14.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.724+0000 7f155b7fe640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:14.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.724+0000 7f155b7fe640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-25T19:26:14.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.724+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f156804f1e0 con 0x7f157413f710 2024-03-25T19:26:14.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.724+0000 7f155b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:14.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.724+0000 7f155b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:14.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.939+0000 7f15794a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:14.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.939+0000 7f15794a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:14.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.939+0000 7f15794a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:14.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.939+0000 7f15794a8640 1 -- 172.21.15.123:0/1621550524 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1574141c00 con 0x7f157413f710 2024-03-25T19:26:14.955 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.955+0000 7f155b7fe640 1 -- 172.21.15.123:0/1621550524 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f1568051110 con 0x7f157413f710 2024-03-25T19:26:14.955 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.955+0000 7f155b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:14.955 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.955+0000 7f155b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:14.955 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:26:14.955 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:14.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 10 librados: watch_flush enter 2024-03-25T19:26:14.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 10 librados: watch_flush exit 2024-03-25T19:26:14.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-25T19:26:14.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 10 client.4377.objecter successfully canceled tick 2024-03-25T19:26:14.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 1 -- 172.21.15.123:0/1621550524 >> v1:172.21.15.151:6800/727684239 conn(0x7f156006e0b0 legacy=0x7f1560070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:14.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 10 monclient: shutdown 2024-03-25T19:26:14.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:14.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 1 -- 172.21.15.123:0/1621550524 >> v1:172.21.15.123:6789/0 conn(0x7f157413f710 legacy=0x7f15741d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:14.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 1 -- 172.21.15.123:0/1621550524 shutdown_connections 2024-03-25T19:26:14.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 1 -- 172.21.15.123:0/1621550524 >> 172.21.15.123:0/1621550524 conn(0x7f15740b7ea0 msgr2=0x7f1574074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:14.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 1 -- 172.21.15.123:0/1621550524 shutdown_connections 2024-03-25T19:26:14.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 1 -- 172.21.15.123:0/1621550524 wait complete. 2024-03-25T19:26:14.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:14.957+0000 7f15794a8640 1 librados: shutdown 2024-03-25T19:26:14.969 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:21.775 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:21.875 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.875+0000 7f9b593af640 10 monclient: get_monmap_and_config 2024-03-25T19:26:21.875 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.875+0000 7f9b593af640 10 monclient: build_initial_monmap 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.875+0000 7f9b593af640 10 monclient: monmap: 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:26:21.876736+0000 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:26:21.876736+0000 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:26:21.876 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.876+0000 7f9b593af640 1 Processor -- start 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.876+0000 7f9b593af640 1 -- start start 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.876+0000 7f9b593af640 10 monclient: init 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.877+0000 7f9b593af640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.877+0000 7f9b593af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.877+0000 7f9b593af640 10 monclient(hunting): picked mon.noname-c con 0x7f9b540b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.877+0000 7f9b593af640 10 monclient(hunting): picked mon.noname-a con 0x7f9b540b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.877+0000 7f9b593af640 10 monclient(hunting): picked mon.noname-b con 0x7f9b5414be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.877+0000 7f9b593af640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b540af940 con 0x7f9b540b1730 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.877+0000 7f9b593af640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b540b0a70 con 0x7f9b540b4d60 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.877+0000 7f9b593af640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b540b0d50 con 0x7f9b5414be90 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.877+0000 7f9b593af640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:21.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.877+0000 7f9b593af640 10 monclient(hunting): authenticate will time out at 5819.237272s 2024-03-25T19:26:21.887 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.887+0000 7f9b52ffd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f9b540b4d60 0x7f9b5413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:49660/0 (socket says 172.21.15.123:49660) 2024-03-25T19:26:21.887 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.887+0000 7f9b52ffd640 1 -- 172.21.15.123:0/2167473685 learned_addr learned my addr 172.21.15.123:0/2167473685 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:21.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.902+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249180420 0 0) 0x7f9b540b0a70 con 0x7f9b540b4d60 2024-03-25T19:26:21.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.902+0000 7f9b527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:21.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.902+0000 7f9b527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:21.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.902+0000 7f9b527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:21.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.902+0000 7f9b527fc640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:26:21.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.902+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b40003520 con 0x7f9b540b4d60 2024-03-25T19:26:21.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572259872 0 0) 0x7f9b540b0d50 con 0x7f9b5414be90 2024-03-25T19:26:21.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:21.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:21.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:21.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:26:21.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b540b0a70 con 0x7f9b5414be90 2024-03-25T19:26:21.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613250356 0 0) 0x7f9b540af940 con 0x7f9b540b1730 2024-03-25T19:26:21.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:21.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:21.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:21.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:26:21.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.907+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b540b0d50 con 0x7f9b540b1730 2024-03-25T19:26:21.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3183569352 0 0) 0x7f9b540b0a70 con 0x7f9b5414be90 2024-03-25T19:26:21.920 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 >> v1:172.21.15.151:6789/0 conn(0x7f9b540b4d60 legacy=0x7f9b5413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 >> v1:172.21.15.123:6789/0 conn(0x7f9b540b1730 legacy=0x7f9b540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 10 monclient: _finish_hunting 0 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 1 monclient: found mon.noname-b 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b540b1030 con 0x7f9b5414be90 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 10 monclient: _finish_auth 0 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 10 monclient: _check_auth_tickets 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b4c0019f0 con 0x7f9b5414be90 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b593af640 5 monclient: authenticate success, global_id 4403 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b593af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:21.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b527fc640 10 monclient: dump: 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:21.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.920+0000 7f9b593af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:21.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.945+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b4c001b50 con 0x7f9b5414be90 2024-03-25T19:26:21.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.945+0000 7f9b527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:21.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.945+0000 7f9b527fc640 1 -- 172.21.15.123:0/2167473685 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b4c001f60 con 0x7f9b5414be90 2024-03-25T19:26:21.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.945+0000 7f9b593af640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:21.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.945+0000 7f9b593af640 4 set_mon_vals no callback set 2024-03-25T19:26:21.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.945+0000 7f9b527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:21.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.945+0000 7f9b527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:21.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.945+0000 7f9b527fc640 10 monclient: dump: 2024-03-25T19:26:21.946 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:21.946 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:21.946 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:21.946 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:21.946 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:21.946 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:21.946 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:21.946 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:21.946 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:21.946 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:21.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.946+0000 7f9b593af640 10 monclient: shutdown 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.946+0000 7f9b593af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.946+0000 7f9b593af640 1 -- 172.21.15.123:0/2167473685 >> v1:172.21.15.123:6790/0 conn(0x7f9b5414be90 legacy=0x7f9b5414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.946+0000 7f9b593af640 1 -- 172.21.15.123:0/2167473685 shutdown_connections 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.946+0000 7f9b593af640 1 -- 172.21.15.123:0/2167473685 >> 172.21.15.123:0/2167473685 conn(0x7f9b540617a0 msgr2=0x7f9b54061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.946+0000 7f9b593af640 1 -- 172.21.15.123:0/2167473685 shutdown_connections 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.946+0000 7f9b593af640 1 -- 172.21.15.123:0/2167473685 wait complete. 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.947+0000 7f9b593af640 10 monclient: build_initial_monmap 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.947+0000 7f9b593af640 10 monclient: monmap: 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:26:21.948 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.947+0000 7f9b593af640 1 librados: starting msgr at 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.947+0000 7f9b593af640 1 librados: starting objecter 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.947+0000 7f9b593af640 1 Processor -- start 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 1 -- start start 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 1 librados: setting wanted keys 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 1 librados: calling monclient init 2024-03-25T19:26:21.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 10 monclient: init 2024-03-25T19:26:21.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:21.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:21.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 10 monclient(hunting): picked mon.noname-c con 0x7f9b540b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:21.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 10 monclient(hunting): picked mon.noname-b con 0x7f9b540b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:21.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 10 monclient(hunting): picked mon.noname-a con 0x7f9b5414be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:21.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b541db2a0 con 0x7f9b5414be90 2024-03-25T19:26:21.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b541dbe40 con 0x7f9b540b4d60 2024-03-25T19:26:21.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b541daec0 con 0x7f9b540b1730 2024-03-25T19:26:21.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:21.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.948+0000 7f9b593af640 10 monclient(hunting): authenticate will time out at 5819.308615s 2024-03-25T19:26:21.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.958+0000 7f9b52ffd640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f9b540b1730 0x7f9b541db730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:45560/0 (socket says 172.21.15.123:45560) 2024-03-25T19:26:21.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.959+0000 7f9b52ffd640 1 -- 172.21.15.123:0/1749004538 learned_addr learned my addr 172.21.15.123:0/1749004538 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:21.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.962+0000 7f9b4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9b540b1730 2024-03-25T19:26:21.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.962+0000 7f9b4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:21.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.963+0000 7f9b4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9b5414be90 2024-03-25T19:26:21.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.963+0000 7f9b4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:21.970 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.971+0000 7f9b4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9b540b4d60 2024-03-25T19:26:21.971 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.971+0000 7f9b4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:21.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.975+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845364948 0 0) 0x7f9b541daec0 con 0x7f9b540b1730 2024-03-25T19:26:21.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.975+0000 7f9b4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:21.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.975+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:21.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.975+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:21.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.975+0000 7f9b4b7fe640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:26:21.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.975+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b34004eb0 con 0x7f9b540b1730 2024-03-25T19:26:21.980 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.981+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187010630 0 0) 0x7f9b541dbe40 con 0x7f9b540b4d60 2024-03-25T19:26:21.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.981+0000 7f9b4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:21.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.981+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:21.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.981+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:21.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.981+0000 7f9b4b7fe640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:26:21.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.981+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b541daec0 con 0x7f9b540b4d60 2024-03-25T19:26:21.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.984+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992156861 0 0) 0x7f9b541db2a0 con 0x7f9b5414be90 2024-03-25T19:26:21.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.984+0000 7f9b4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:21.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.984+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:21.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.984+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:21.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.984+0000 7f9b4b7fe640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:26:21.984 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.984+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b541dbe40 con 0x7f9b5414be90 2024-03-25T19:26:21.997 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.997+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4053949820 0 0) 0x7f9b541daec0 con 0x7f9b540b4d60 2024-03-25T19:26:21.997 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.997+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b541db2a0 con 0x7f9b540b4d60 2024-03-25T19:26:21.997 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.997+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b3c001870 con 0x7f9b540b4d60 2024-03-25T19:26:21.997 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.997+0000 7f9b4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.997+0000 7f9b4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.997+0000 7f9b4b7fe640 10 monclient(hunting): dump: 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:21.998 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:21.999 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:21.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:21.997+0000 7f9b4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:22.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.000+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3904328106 0 0) 0x7f9b34004eb0 con 0x7f9b540b1730 2024-03-25T19:26:22.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.000+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b541daec0 con 0x7f9b540b1730 2024-03-25T19:26:22.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.000+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b44001950 con 0x7f9b540b1730 2024-03-25T19:26:22.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.000+0000 7f9b4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:22.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.000+0000 7f9b4b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:22.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.000+0000 7f9b4b7fe640 10 monclient(hunting): dump: 2024-03-25T19:26:22.001 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:22.001 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:22.001 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:22.001 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:22.001 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:22.001 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:22.001 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:22.001 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:22.001 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:22.001 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3568823574 0 0) 0x7f9b541db2a0 con 0x7f9b540b4d60 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 >> v1:172.21.15.123:6790/0 conn(0x7f9b540b1730 legacy=0x7f9b541db730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 >> v1:172.21.15.123:6789/0 conn(0x7f9b5414be90 legacy=0x7f9b54139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 1 monclient: found mon.b 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b541334a0 con 0x7f9b540b4d60 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:26:22.019 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:22.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b593af640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:26:22.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b593af640 10 monclient: _renew_subs 2024-03-25T19:26:22.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b593af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:22.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b593af640 1 -- 172.21.15.123:0/1749004538 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b541336f0 con 0x7f9b540b4d60 2024-03-25T19:26:22.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b593af640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:22.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b593af640 10 monclient: _renew_subs 2024-03-25T19:26:22.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b593af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:22.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b593af640 1 -- 172.21.15.123:0/1749004538 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5413d2d0 con 0x7f9b540b4d60 2024-03-25T19:26:22.020 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.018+0000 7f9b593af640 1 librados: init done 2024-03-25T19:26:22.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.019+0000 7f9b593af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:22.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.019+0000 7f9b593af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:22.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.019+0000 7f9b593af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:22.021 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.019+0000 7f9b593af640 1 -- 172.21.15.123:0/1749004538 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b540627b0 con 0x7f9b540b4d60 2024-03-25T19:26:22.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.327+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b3c003610 con 0x7f9b540b4d60 2024-03-25T19:26:22.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.327+0000 7f9b4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:22.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.327+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9b3c001d80 con 0x7f9b540b4d60 2024-03-25T19:26:22.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.328+0000 7f9b4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:22.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.328+0000 7f9b4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:22.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.328+0000 7f9b4b7fe640 10 monclient: dump: 2024-03-25T19:26:22.328 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:22.328 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:22.328 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:22.328 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:22.329 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:22.329 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:22.329 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:22.329 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:22.329 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:22.329 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:22.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.328+0000 7f9b51ffb640 4 set_mon_vals no callback set 2024-03-25T19:26:22.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.329+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f9b3c003b20 con 0x7f9b540b4d60 2024-03-25T19:26:22.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.329+0000 7f9b52ffd640 10 monclient: get_auth_request con 0x7f9b3406df10 auth_method 0 2024-03-25T19:26:22.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.330+0000 7f9b4b7fe640 10 client.4396.objecter ms_handle_connect 0x7f9b3406df10 2024-03-25T19:26:22.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.333+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f9b3c0535c0 con 0x7f9b540b4d60 2024-03-25T19:26:22.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.333+0000 7f9b4b7fe640 10 client.4396.objecter ms_dispatch 0x7f9b54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:22.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.333+0000 7f9b4b7fe640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:22.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.333+0000 7f9b4b7fe640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:22.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.333+0000 7f9b4b7fe640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T19:26:22.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.333+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b3c04f0d0 con 0x7f9b540b4d60 2024-03-25T19:26:22.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.333+0000 7f9b4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:22.333 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.333+0000 7f9b4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:22.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.582+0000 7f9b593af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:22.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.582+0000 7f9b593af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:22.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.582+0000 7f9b593af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:22.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.582+0000 7f9b593af640 1 -- 172.21.15.123:0/1749004538 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b541db050 con 0x7f9b540b4d60 2024-03-25T19:26:22.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.628+0000 7f9b4b7fe640 1 -- 172.21.15.123:0/1749004538 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f9b3c051000 con 0x7f9b540b4d60 2024-03-25T19:26:22.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.628+0000 7f9b4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:22.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.628+0000 7f9b4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:22.629 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:26:22.629 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 10 librados: watch_flush enter 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 10 librados: watch_flush exit 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 10 client.4396.objecter successfully canceled tick 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 1 -- 172.21.15.123:0/1749004538 >> v1:172.21.15.151:6800/727684239 conn(0x7f9b3406df10 legacy=0x7f9b340703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 10 monclient: shutdown 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 1 -- 172.21.15.123:0/1749004538 >> v1:172.21.15.151:6789/0 conn(0x7f9b540b4d60 legacy=0x7f9b54132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 1 -- 172.21.15.123:0/1749004538 shutdown_connections 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 1 -- 172.21.15.123:0/1749004538 >> 172.21.15.123:0/1749004538 conn(0x7f9b540617a0 msgr2=0x7f9b540b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 1 -- 172.21.15.123:0/1749004538 shutdown_connections 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 1 -- 172.21.15.123:0/1749004538 wait complete. 2024-03-25T19:26:22.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:22.631+0000 7f9b593af640 1 librados: shutdown 2024-03-25T19:26:22.643 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:29.450 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:29.566 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.566+0000 7f7e5ae47640 10 monclient: get_monmap_and_config 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.566+0000 7f7e5ae47640 10 monclient: build_initial_monmap 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.566+0000 7f7e5ae47640 10 monclient: monmap: 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:26:29.567792+0000 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:26:29.567792+0000 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:29.568 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:26:29.568 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:26:29.568 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:29.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.567+0000 7f7e5ae47640 1 Processor -- start 2024-03-25T19:26:29.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.568+0000 7f7e5ae47640 1 -- start start 2024-03-25T19:26:29.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.568+0000 7f7e5ae47640 10 monclient: init 2024-03-25T19:26:29.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.569+0000 7f7e5ae47640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:29.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.569+0000 7f7e5ae47640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:26:29.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.569+0000 7f7e5ae47640 10 monclient(hunting): picked mon.noname-a con 0x7f7e540a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:29.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.569+0000 7f7e5ae47640 10 monclient(hunting): picked mon.noname-c con 0x7f7e5413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:29.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.569+0000 7f7e5ae47640 10 monclient(hunting): picked mon.noname-b con 0x7f7e541430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:29.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.569+0000 7f7e5ae47640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e54142520 con 0x7f7e540a98f0 2024-03-25T19:26:29.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.569+0000 7f7e5ae47640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e540b28a0 con 0x7f7e5413f710 2024-03-25T19:26:29.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.569+0000 7f7e5ae47640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e540b2a30 con 0x7f7e541430c0 2024-03-25T19:26:29.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.569+0000 7f7e5ae47640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:29.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.569+0000 7f7e5ae47640 10 monclient(hunting): authenticate will time out at 5826.929310s 2024-03-25T19:26:29.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.573+0000 7f7e53fff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f7e5413f710 0x7f7e541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:48252/0 (socket says 172.21.15.123:48252) 2024-03-25T19:26:29.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.573+0000 7f7e53fff640 1 -- 172.21.15.123:0/2906554177 learned_addr learned my addr 172.21.15.123:0/2906554177 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:29.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.587+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450229979 0 0) 0x7f7e540b28a0 con 0x7f7e5413f710 2024-03-25T19:26:29.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.587+0000 7f7e537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.587+0000 7f7e537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.587+0000 7f7e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.587+0000 7f7e537fe640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:26:29.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.587+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e44003520 con 0x7f7e5413f710 2024-03-25T19:26:29.591 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.591+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403776363 0 0) 0x7f7e54142520 con 0x7f7e540a98f0 2024-03-25T19:26:29.591 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.591+0000 7f7e537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.591 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.591+0000 7f7e537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.591 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.591+0000 7f7e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.591 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.591+0000 7f7e537fe640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:26:29.591 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.591+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e540b28a0 con 0x7f7e540a98f0 2024-03-25T19:26:29.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.600+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4105986662 0 0) 0x7f7e540b28a0 con 0x7f7e540a98f0 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.600+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 >> v1:172.21.15.123:6790/0 conn(0x7f7e541430c0 legacy=0x7f7e5414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.600+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 >> v1:172.21.15.151:6789/0 conn(0x7f7e5413f710 legacy=0x7f7e541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.600+0000 7f7e537fe640 10 monclient: _finish_hunting 0 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.600+0000 7f7e537fe640 1 monclient: found mon.noname-a 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e537fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e540b44c0 con 0x7f7e540a98f0 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e537fe640 10 monclient: _finish_auth 0 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e537fe640 10 monclient: _check_auth_tickets 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7e48001670 con 0x7f7e540a98f0 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e5ae47640 5 monclient: authenticate success, global_id 4386 2024-03-25T19:26:29.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e5ae47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e537fe640 10 monclient: dump: 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:29.602 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:29.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.601+0000 7f7e5ae47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.610+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e480017d0 con 0x7f7e540a98f0 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.610+0000 7f7e537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.610+0000 7f7e537fe640 1 -- 172.21.15.123:0/2906554177 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7e48001ba0 con 0x7f7e540a98f0 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.610+0000 7f7e537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.610+0000 7f7e537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.610+0000 7f7e537fe640 10 monclient: dump: 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.610 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.611 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:29.611 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:29.611 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:29.611 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:29.611 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:29.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.610+0000 7f7e5ae47640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:29.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.610+0000 7f7e5ae47640 4 set_mon_vals no callback set 2024-03-25T19:26:29.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.611+0000 7f7e5ae47640 10 monclient: shutdown 2024-03-25T19:26:29.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.611+0000 7f7e5ae47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:29.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.611+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2906554177 >> v1:172.21.15.123:6789/0 conn(0x7f7e540a98f0 legacy=0x7f7e540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.611+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2906554177 shutdown_connections 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.611+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2906554177 >> 172.21.15.123:0/2906554177 conn(0x7f7e540b7ea0 msgr2=0x7f7e540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.611+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2906554177 shutdown_connections 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.611+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2906554177 wait complete. 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.612+0000 7f7e5ae47640 10 monclient: build_initial_monmap 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.612+0000 7f7e5ae47640 10 monclient: monmap: 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:29.613 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:26:29.614 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:26:29.614 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:29.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.612+0000 7f7e5ae47640 1 librados: starting msgr at 2024-03-25T19:26:29.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.612+0000 7f7e5ae47640 1 librados: starting objecter 2024-03-25T19:26:29.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.612+0000 7f7e5ae47640 1 Processor -- start 2024-03-25T19:26:29.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.612+0000 7f7e5ae47640 1 -- start start 2024-03-25T19:26:29.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.612+0000 7f7e5ae47640 1 librados: setting wanted keys 2024-03-25T19:26:29.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.612+0000 7f7e5ae47640 1 librados: calling monclient init 2024-03-25T19:26:29.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.612+0000 7f7e5ae47640 10 monclient: init 2024-03-25T19:26:29.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.613+0000 7f7e5ae47640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:29.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.613+0000 7f7e5ae47640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:26:29.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.613+0000 7f7e5ae47640 10 monclient(hunting): picked mon.noname-c con 0x7f7e540a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:29.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.613+0000 7f7e5ae47640 10 monclient(hunting): picked mon.noname-a con 0x7f7e5413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:29.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.613+0000 7f7e5ae47640 10 monclient(hunting): picked mon.noname-b con 0x7f7e541430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:29.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.613+0000 7f7e5ae47640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5413b310 con 0x7f7e5413f710 2024-03-25T19:26:29.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.613+0000 7f7e5ae47640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5413b580 con 0x7f7e541430c0 2024-03-25T19:26:29.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.613+0000 7f7e5ae47640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e540b7a20 con 0x7f7e540a98f0 2024-03-25T19:26:29.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.613+0000 7f7e5ae47640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:29.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.613+0000 7f7e5ae47640 10 monclient(hunting): authenticate will time out at 5826.973616s 2024-03-25T19:26:29.624 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.624+0000 7f7e53fff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f7e5413f710 0x7f7e5412f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:40666/0 (socket says 172.21.15.123:40666) 2024-03-25T19:26:29.624 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.624+0000 7f7e53fff640 1 -- 172.21.15.123:0/2266292761 learned_addr learned my addr 172.21.15.123:0/2266292761 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:29.626 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.626+0000 7f7e517fa640 10 client.?.objecter ms_handle_connect 0x7f7e5413f710 2024-03-25T19:26:29.626 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.626+0000 7f7e517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:29.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.628+0000 7f7e517fa640 10 client.?.objecter ms_handle_connect 0x7f7e540a98f0 2024-03-25T19:26:29.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.628+0000 7f7e517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:29.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.628+0000 7f7e517fa640 10 client.?.objecter ms_handle_connect 0x7f7e541430c0 2024-03-25T19:26:29.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.628+0000 7f7e517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:29.641 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.641+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525193974 0 0) 0x7f7e5413b310 con 0x7f7e5413f710 2024-03-25T19:26:29.641 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.641+0000 7f7e517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.641 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.641+0000 7f7e517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.641 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.641+0000 7f7e517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.641 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.641+0000 7f7e517fa640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:26:29.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.641+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e38004eb0 con 0x7f7e5413f710 2024-03-25T19:26:29.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.644+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981324969 0 0) 0x7f7e540b7a20 con 0x7f7e540a98f0 2024-03-25T19:26:29.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.644+0000 7f7e517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.644+0000 7f7e517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.644+0000 7f7e517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.644+0000 7f7e517fa640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:26:29.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.644+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e5413b310 con 0x7f7e540a98f0 2024-03-25T19:26:29.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.645+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148465212 0 0) 0x7f7e5413b580 con 0x7f7e541430c0 2024-03-25T19:26:29.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.645+0000 7f7e517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.645+0000 7f7e517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.645+0000 7f7e517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.645+0000 7f7e517fa640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:26:29.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.645+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e540b7a20 con 0x7f7e541430c0 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.651+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1224064271 0 0) 0x7f7e38004eb0 con 0x7f7e5413f710 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.651+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e5413b580 con 0x7f7e5413f710 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.651+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7e40001990 con 0x7f7e5413f710 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.651+0000 7f7e517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.651+0000 7f7e517fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.651+0000 7f7e517fa640 10 monclient(hunting): dump: 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.651 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.652 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:29.652 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:29.652 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:29.652 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:29.652 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:29.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.651+0000 7f7e517fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:29.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.659+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2188811247 0 0) 0x7f7e5413b310 con 0x7f7e540a98f0 2024-03-25T19:26:29.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.659+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e38004eb0 con 0x7f7e540a98f0 2024-03-25T19:26:29.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.659+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7e48001ae0 con 0x7f7e540a98f0 2024-03-25T19:26:29.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.659+0000 7f7e517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.659+0000 7f7e517fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:29.659 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.659+0000 7f7e517fa640 10 monclient(hunting): dump: 2024-03-25T19:26:29.660 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:29.660 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:29.660 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.660 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.660 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.660 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:29.660 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:29.660 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:29.660 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:29.660 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:29.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.662+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3322042491 0 0) 0x7f7e540b7a20 con 0x7f7e541430c0 2024-03-25T19:26:29.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.662+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e5413b310 con 0x7f7e541430c0 2024-03-25T19:26:29.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.662+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7e4c0019e0 con 0x7f7e541430c0 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.662+0000 7f7e517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.662+0000 7f7e517fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.662+0000 7f7e517fa640 10 monclient(hunting): dump: 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:29.663 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1473675248 0 0) 0x7f7e5413b580 con 0x7f7e5413f710 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 >> v1:172.21.15.123:6790/0 conn(0x7f7e540a98f0 legacy=0x7f7e5413b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 >> v1:172.21.15.151:6789/0 conn(0x7f7e541430c0 legacy=0x7f7e54139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 10 monclient: _finish_hunting 0 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 1 monclient: found mon.a 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e540b7640 con 0x7f7e5413f710 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 10 monclient: _finish_auth 0 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 10 monclient: _check_auth_tickets 2024-03-25T19:26:29.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:29.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e5ae47640 5 monclient: authenticate success, global_id 4389 2024-03-25T19:26:29.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e5ae47640 10 monclient: _renew_subs 2024-03-25T19:26:29.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e5ae47640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:29.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.669+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e54139780 con 0x7f7e5413f710 2024-03-25T19:26:29.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.670+0000 7f7e5ae47640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:29.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.670+0000 7f7e5ae47640 10 monclient: _renew_subs 2024-03-25T19:26:29.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.670+0000 7f7e5ae47640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:29.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.670+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e541e3480 con 0x7f7e5413f710 2024-03-25T19:26:29.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.670+0000 7f7e5ae47640 1 librados: init done 2024-03-25T19:26:29.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.671+0000 7f7e5ae47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:29.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.671+0000 7f7e5ae47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:29.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.671+0000 7f7e5ae47640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:29.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.671+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e540627b0 con 0x7f7e5413f710 2024-03-25T19:26:29.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.989+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e400047e0 con 0x7f7e5413f710 2024-03-25T19:26:29.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.989+0000 7f7e517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:29.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.989+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7e40004af0 con 0x7f7e5413f710 2024-03-25T19:26:29.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.989+0000 7f7e517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.989+0000 7f7e517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.989+0000 7f7e517fa640 10 monclient: dump: 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:29.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.989+0000 7f7e52ffd640 4 set_mon_vals no callback set 2024-03-25T19:26:29.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.990+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f7e4001ab40 con 0x7f7e5413f710 2024-03-25T19:26:29.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.991+0000 7f7e58be3640 10 monclient: get_auth_request con 0x7f7e3806e0f0 auth_method 0 2024-03-25T19:26:29.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.991+0000 7f7e517fa640 10 client.4389.objecter ms_handle_connect 0x7f7e3806e0f0 2024-03-25T19:26:29.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.996+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f7e400548c0 con 0x7f7e5413f710 2024-03-25T19:26:29.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.996+0000 7f7e517fa640 10 client.4389.objecter ms_dispatch 0x7f7e54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:29.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.996+0000 7f7e517fa640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:29.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.996+0000 7f7e517fa640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:29.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.996+0000 7f7e517fa640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-25T19:26:29.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.996+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e40050200 con 0x7f7e5413f710 2024-03-25T19:26:29.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.996+0000 7f7e517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:29.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:29.996+0000 7f7e517fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:30.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.239+0000 7f7e5ae47640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:30.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.239+0000 7f7e5ae47640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:30.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.239+0000 7f7e5ae47640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:30.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.239+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2266292761 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e5413d7c0 con 0x7f7e5413f710 2024-03-25T19:26:30.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.312+0000 7f7e517fa640 1 -- 172.21.15.123:0/2266292761 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f7e40052300 con 0x7f7e5413f710 2024-03-25T19:26:30.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.312+0000 7f7e517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:30.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.312+0000 7f7e517fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:30.313 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:26:30.313 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:30.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.314+0000 7f7e5ae47640 10 librados: watch_flush enter 2024-03-25T19:26:30.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.314+0000 7f7e5ae47640 10 librados: watch_flush exit 2024-03-25T19:26:30.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.314+0000 7f7e5ae47640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-25T19:26:30.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.314+0000 7f7e5ae47640 10 client.4389.objecter successfully canceled tick 2024-03-25T19:26:30.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.314+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2266292761 >> v1:172.21.15.151:6800/727684239 conn(0x7f7e3806e0f0 legacy=0x7f7e380705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:30.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.314+0000 7f7e5ae47640 10 monclient: shutdown 2024-03-25T19:26:30.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.314+0000 7f7e5ae47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:30.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.314+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2266292761 >> v1:172.21.15.123:6789/0 conn(0x7f7e5413f710 legacy=0x7f7e5412f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:30.316 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.314+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2266292761 shutdown_connections 2024-03-25T19:26:30.316 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.314+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2266292761 >> 172.21.15.123:0/2266292761 conn(0x7f7e540b7ea0 msgr2=0x7f7e54074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:30.316 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.315+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2266292761 shutdown_connections 2024-03-25T19:26:30.316 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.315+0000 7f7e5ae47640 1 -- 172.21.15.123:0/2266292761 wait complete. 2024-03-25T19:26:30.316 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:30.315+0000 7f7e5ae47640 1 librados: shutdown 2024-03-25T19:26:30.324 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:37.130 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:37.244 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.245+0000 7ff100a27640 10 monclient: get_monmap_and_config 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.245+0000 7ff100a27640 10 monclient: build_initial_monmap 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.245+0000 7ff100a27640 10 monclient: monmap: 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:26:37.245975+0000 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:26:37.245975+0000 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:26:37.245 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:37.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.245+0000 7ff100a27640 1 Processor -- start 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.245+0000 7ff100a27640 1 -- start start 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.245+0000 7ff100a27640 10 monclient: init 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.246+0000 7ff100a27640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.246+0000 7ff100a27640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.246+0000 7ff100a27640 10 monclient(hunting): picked mon.noname-a con 0x7ff0fc0a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.246+0000 7ff100a27640 10 monclient(hunting): picked mon.noname-c con 0x7ff0fc13f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.246+0000 7ff100a27640 10 monclient(hunting): picked mon.noname-b con 0x7ff0fc1430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.246+0000 7ff100a27640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0fc142520 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.246+0000 7ff100a27640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0fc0b28a0 con 0x7ff0fc13f710 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.246+0000 7ff100a27640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0fc0b2a30 con 0x7ff0fc1430c0 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.246+0000 7ff100a27640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:37.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.246+0000 7ff100a27640 10 monclient(hunting): authenticate will time out at 5834.606657s 2024-03-25T19:26:37.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.258+0000 7ff0fa59c640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff0fc0a98f0 0x7ff0fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:35908/0 (socket says 172.21.15.123:35908) 2024-03-25T19:26:37.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.258+0000 7ff0fa59c640 1 -- 172.21.15.123:0/3951215387 learned_addr learned my addr 172.21.15.123:0/3951215387 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:37.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.268+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358798741 0 0) 0x7ff0fc142520 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.268+0000 7ff0f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:26:37.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.268+0000 7ff0f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:37.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.268+0000 7ff0f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:37.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.268+0000 7ff0f959a640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:26:37.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.268+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e4003520 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.276 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.276+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159280219 0 0) 0x7ff0fc0b28a0 con 0x7ff0fc13f710 2024-03-25T19:26:37.276 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.276+0000 7ff0f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:26:37.276 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.276+0000 7ff0f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:37.276 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.276+0000 7ff0f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:37.276 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.276+0000 7ff0f959a640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:26:37.276 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.276+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0fc142520 con 0x7ff0fc13f710 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1990909644 0 0) 0x7ff0e4003520 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 >> v1:172.21.15.123:6790/0 conn(0x7ff0fc1430c0 legacy=0x7ff0fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 >> v1:172.21.15.151:6789/0 conn(0x7ff0fc13f710 legacy=0x7ff0fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 10 monclient: _finish_hunting 0 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 1 monclient: found mon.noname-a 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc0b44c0 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 10 monclient: _finish_auth 0 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 10 monclient: _check_auth_tickets 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:37.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff0ec001690 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff0f959a640 10 monclient: dump: 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:37.280 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:37.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff100a27640 5 monclient: authenticate success, global_id 4392 2024-03-25T19:26:37.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.278+0000 7ff100a27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:37.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.282+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0ec0017f0 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.282+0000 7ff0f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:37.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.282+0000 7ff0f959a640 1 -- 172.21.15.123:0/3951215387 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff0ec001c00 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.282+0000 7ff100a27640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.282+0000 7ff100a27640 4 set_mon_vals no callback set 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.282+0000 7ff0f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.282+0000 7ff0f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.282+0000 7ff0f959a640 10 monclient: dump: 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:37.283 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:37.285 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.283+0000 7ff100a27640 10 monclient: shutdown 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.283+0000 7ff100a27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.283+0000 7ff100a27640 1 -- 172.21.15.123:0/3951215387 >> v1:172.21.15.123:6789/0 conn(0x7ff0fc0a98f0 legacy=0x7ff0fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.283+0000 7ff100a27640 1 -- 172.21.15.123:0/3951215387 shutdown_connections 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.283+0000 7ff100a27640 1 -- 172.21.15.123:0/3951215387 >> 172.21.15.123:0/3951215387 conn(0x7ff0fc0b7ea0 msgr2=0x7ff0fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.283+0000 7ff100a27640 1 -- 172.21.15.123:0/3951215387 shutdown_connections 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.283+0000 7ff100a27640 1 -- 172.21.15.123:0/3951215387 wait complete. 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.284+0000 7ff100a27640 10 monclient: build_initial_monmap 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.284+0000 7ff100a27640 10 monclient: monmap: 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:37.286 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.284+0000 7ff100a27640 1 librados: starting msgr at 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.284+0000 7ff100a27640 1 librados: starting objecter 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.284+0000 7ff100a27640 1 Processor -- start 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 1 -- start start 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 1 librados: setting wanted keys 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 1 librados: calling monclient init 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 10 monclient: init 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:37.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:37.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 10 monclient(hunting): picked mon.noname-c con 0x7ff0fc0a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:37.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 10 monclient(hunting): picked mon.noname-b con 0x7ff0fc13f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:37.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 10 monclient(hunting): picked mon.noname-a con 0x7ff0fc1430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:37.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0fc138c70 con 0x7ff0fc1430c0 2024-03-25T19:26:37.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0fc1367d0 con 0x7ff0fc13f710 2024-03-25T19:26:37.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0fc136960 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:37.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.285+0000 7ff100a27640 10 monclient(hunting): authenticate will time out at 5834.645534s 2024-03-25T19:26:37.292 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.292+0000 7ff0fad9d640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff0fc1430c0 0x7ff0fc1e2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:35950/0 (socket says 172.21.15.123:35950) 2024-03-25T19:26:37.292 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.292+0000 7ff0fad9d640 1 -- 172.21.15.123:0/1049784576 learned_addr learned my addr 172.21.15.123:0/1049784576 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:37.292 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.292+0000 7ff0eaffd640 10 client.?.objecter ms_handle_connect 0x7ff0fc1430c0 2024-03-25T19:26:37.292 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.292+0000 7ff0eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:37.296 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.296+0000 7ff0eaffd640 10 client.?.objecter ms_handle_connect 0x7ff0fc13f710 2024-03-25T19:26:37.296 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.296+0000 7ff0eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:37.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.304+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496223979 0 0) 0x7ff0fc1367d0 con 0x7ff0fc13f710 2024-03-25T19:26:37.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.304+0000 7ff0eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:26:37.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.304+0000 7ff0eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:37.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.304+0000 7ff0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:37.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.304+0000 7ff0eaffd640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:26:37.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.304+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0dc004eb0 con 0x7ff0fc13f710 2024-03-25T19:26:37.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.305+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715609953 0 0) 0x7ff0fc138c70 con 0x7ff0fc1430c0 2024-03-25T19:26:37.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.305+0000 7ff0eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:26:37.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.305+0000 7ff0eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:37.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.305+0000 7ff0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:37.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.305+0000 7ff0eaffd640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:26:37.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.305+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0fc1367d0 con 0x7ff0fc1430c0 2024-03-25T19:26:37.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.312+0000 7ff0eaffd640 10 client.?.objecter ms_handle_connect 0x7ff0fc0a98f0 2024-03-25T19:26:37.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.312+0000 7ff0eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:37.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.316+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 45232751 0 0) 0x7ff0dc004eb0 con 0x7ff0fc13f710 2024-03-25T19:26:37.316 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.316+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0fc138c70 con 0x7ff0fc13f710 2024-03-25T19:26:37.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.317+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2178478839 0 0) 0x7ff0fc1367d0 con 0x7ff0fc1430c0 2024-03-25T19:26:37.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.317+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0dc004eb0 con 0x7ff0fc1430c0 2024-03-25T19:26:37.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.318+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff0f00019f0 con 0x7ff0fc13f710 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.318+0000 7ff0eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.318+0000 7ff0eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.318+0000 7ff0eaffd640 10 monclient(hunting): dump: 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:37.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.318+0000 7ff0eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:37.324 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.324+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff0f4001c90 con 0x7ff0fc1430c0 2024-03-25T19:26:37.324 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.324+0000 7ff0eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:37.324 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.324+0000 7ff0eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:37.324 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.324+0000 7ff0eaffd640 10 monclient(hunting): dump: 2024-03-25T19:26:37.324 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:37.324 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:37.325 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:37.325 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:37.325 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:37.325 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:37.325 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:37.325 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:37.325 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:37.325 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451557433 0 0) 0x7ff0fc136960 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0fc1367d0 con 0x7ff0fc0a98f0 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2939779273 0 0) 0x7ff0fc138c70 con 0x7ff0fc13f710 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 >> v1:172.21.15.123:6790/0 conn(0x7ff0fc0a98f0 legacy=0x7ff0fc137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 >> v1:172.21.15.123:6789/0 conn(0x7ff0fc1430c0 legacy=0x7ff0fc1e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 1 monclient: found mon.b 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:37.327 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc139630 con 0x7ff0fc13f710 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 10 monclient: _finish_auth 0 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff0eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff100a27640 5 monclient: authenticate success, global_id 4408 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff100a27640 10 monclient: _renew_subs 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff100a27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.325+0000 7ff100a27640 1 -- 172.21.15.123:0/1049784576 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0fc139880 con 0x7ff0fc13f710 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.326+0000 7ff100a27640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.326+0000 7ff100a27640 10 monclient: _renew_subs 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.326+0000 7ff100a27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.326+0000 7ff100a27640 1 -- 172.21.15.123:0/1049784576 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0fc1337f0 con 0x7ff0fc13f710 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.326+0000 7ff100a27640 1 librados: init done 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.327+0000 7ff100a27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.327+0000 7ff100a27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:37.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.327+0000 7ff100a27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:37.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.327+0000 7ff100a27640 1 -- 172.21.15.123:0/1049784576 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0fc0627b0 con 0x7ff0fc13f710 2024-03-25T19:26:37.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.595+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0f0003610 con 0x7ff0fc13f710 2024-03-25T19:26:37.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.595+0000 7ff0eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:37.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.595+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff0f0003920 con 0x7ff0fc13f710 2024-03-25T19:26:37.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.595+0000 7ff0eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:37.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.595+0000 7ff0eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:37.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.595+0000 7ff0eaffd640 10 monclient: dump: 2024-03-25T19:26:37.596 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:37.596 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:37.596 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:37.596 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:37.597 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:37.597 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:37.597 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:37.597 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:37.597 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:37.597 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:37.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.595+0000 7ff0f8d99640 4 set_mon_vals no callback set 2024-03-25T19:26:37.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.596+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7ff0f0019930 con 0x7ff0fc13f710 2024-03-25T19:26:37.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.597+0000 7ff0fa59c640 10 monclient: get_auth_request con 0x7ff0dc06df30 auth_method 0 2024-03-25T19:26:37.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.597+0000 7ff0eaffd640 10 client.4408.objecter ms_handle_connect 0x7ff0dc06df30 2024-03-25T19:26:37.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.602+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7ff0f0053670 con 0x7ff0fc13f710 2024-03-25T19:26:37.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.602+0000 7ff0eaffd640 10 client.4408.objecter ms_dispatch 0x7ff0fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:37.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.602+0000 7ff0eaffd640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:37.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.602+0000 7ff0eaffd640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:37.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.603+0000 7ff0eaffd640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-25T19:26:37.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.603+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0f00b08b0 con 0x7ff0fc13f710 2024-03-25T19:26:37.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.603+0000 7ff0eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:37.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.603+0000 7ff0eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:37.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.860+0000 7ff100a27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:37.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.860+0000 7ff100a27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:37.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.860+0000 7ff100a27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:37.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.860+0000 7ff100a27640 1 -- 172.21.15.123:0/1049784576 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0fc074cc0 con 0x7ff0fc13f710 2024-03-25T19:26:37.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.879+0000 7ff0eaffd640 1 -- 172.21.15.123:0/1049784576 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7ff0f004efb0 con 0x7ff0fc13f710 2024-03-25T19:26:37.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.879+0000 7ff0eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:37.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.879+0000 7ff0eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:37.879 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:26:37.880 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:37.882 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 10 librados: watch_flush enter 2024-03-25T19:26:37.882 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 10 librados: watch_flush exit 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 10 client.4408.objecter successfully canceled tick 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 1 -- 172.21.15.123:0/1049784576 >> v1:172.21.15.151:6800/727684239 conn(0x7ff0dc06df30 legacy=0x7ff0dc0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 10 monclient: shutdown 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 1 -- 172.21.15.123:0/1049784576 >> v1:172.21.15.151:6789/0 conn(0x7ff0fc13f710 legacy=0x7ff0fc132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 1 -- 172.21.15.123:0/1049784576 shutdown_connections 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 1 -- 172.21.15.123:0/1049784576 >> 172.21.15.123:0/1049784576 conn(0x7ff0fc0b7ea0 msgr2=0x7ff0fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 1 -- 172.21.15.123:0/1049784576 shutdown_connections 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 1 -- 172.21.15.123:0/1049784576 wait complete. 2024-03-25T19:26:37.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:37.882+0000 7ff100a27640 1 librados: shutdown 2024-03-25T19:26:37.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:44.700 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.811+0000 7f4517695640 10 monclient: get_monmap_and_config 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.811+0000 7f4517695640 10 monclient: build_initial_monmap 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.811+0000 7f4517695640 10 monclient: monmap: 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:26:44.813361+0000 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:26:44.813361+0000 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:44.812 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:26:44.813 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:26:44.813 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 1 Processor -- start 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 1 -- start start 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 10 monclient: init 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 10 monclient(hunting): picked mon.noname-b con 0x7f45100a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 10 monclient(hunting): picked mon.noname-a con 0x7f451013f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 10 monclient(hunting): picked mon.noname-c con 0x7f45101430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510142520 con 0x7f451013f710 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100b28a0 con 0x7f45101430c0 2024-03-25T19:26:44.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100b2a30 con 0x7f45100a98f0 2024-03-25T19:26:44.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:44.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.812+0000 7f4517695640 10 monclient(hunting): authenticate will time out at 5842.173834s 2024-03-25T19:26:44.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.814+0000 7f4515c32640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f45101430c0 0x7f451014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:39270/0 (socket says 172.21.15.123:39270) 2024-03-25T19:26:44.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.814+0000 7f4515c32640 1 -- 172.21.15.123:0/2379344112 learned_addr learned my addr 172.21.15.123:0/2379344112 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:44.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.894+0000 7f4507fff640 1 -- 172.21.15.123:0/2379344112 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142794201 0 0) 0x7f4510142520 con 0x7f451013f710 2024-03-25T19:26:44.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.894+0000 7f4507fff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:44.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.894+0000 7f4507fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:44.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.894+0000 7f4507fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:44.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.894+0000 7f4507fff640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:26:44.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.894+0000 7f4507fff640 1 -- 172.21.15.123:0/2379344112 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44fc003520 con 0x7f451013f710 2024-03-25T19:26:44.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 1 -- 172.21.15.123:0/2379344112 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288686861 0 0) 0x7f44fc003520 con 0x7f451013f710 2024-03-25T19:26:44.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 1 -- 172.21.15.123:0/2379344112 >> v1:172.21.15.123:6790/0 conn(0x7f45100a98f0 legacy=0x7f45100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:44.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 1 -- 172.21.15.123:0/2379344112 >> v1:172.21.15.151:6789/0 conn(0x7f45101430c0 legacy=0x7f451014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 10 monclient: _finish_hunting 0 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 1 monclient: found mon.noname-a 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 1 -- 172.21.15.123:0/2379344112 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45100b44c0 con 0x7f451013f710 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 10 monclient: _finish_auth 0 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 10 monclient: _check_auth_tickets 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 1 -- 172.21.15.123:0/2379344112 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4500001690 con 0x7f451013f710 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4507fff640 10 monclient: dump: 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:44.904 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:44.905 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:44.905 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:44.905 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:44.905 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:44.905 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:44.905 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:44.905 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:44.905 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4517695640 5 monclient: authenticate success, global_id 4398 2024-03-25T19:26:44.905 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.902+0000 7f4517695640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.912+0000 7f4507fff640 1 -- 172.21.15.123:0/2379344112 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45000017f0 con 0x7f451013f710 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.912+0000 7f4507fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.912+0000 7f4507fff640 1 -- 172.21.15.123:0/2379344112 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4500001c00 con 0x7f451013f710 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.912+0000 7f4517695640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.912+0000 7f4517695640 4 set_mon_vals no callback set 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.912+0000 7f4507fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.912+0000 7f4507fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.912+0000 7f4507fff640 10 monclient: dump: 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:44.913 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:44.914 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:44.914 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:44.914 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:44.914 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:44.914 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:44.914 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:44.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.913+0000 7f4517695640 10 monclient: shutdown 2024-03-25T19:26:44.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.913+0000 7f4517695640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:44.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.913+0000 7f4517695640 1 -- 172.21.15.123:0/2379344112 >> v1:172.21.15.123:6789/0 conn(0x7f451013f710 legacy=0x7f45101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:44.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.913+0000 7f4517695640 1 -- 172.21.15.123:0/2379344112 shutdown_connections 2024-03-25T19:26:44.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.913+0000 7f4517695640 1 -- 172.21.15.123:0/2379344112 >> 172.21.15.123:0/2379344112 conn(0x7f45100b7ea0 msgr2=0x7f45100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:44.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.913+0000 7f4517695640 1 -- 172.21.15.123:0/2379344112 shutdown_connections 2024-03-25T19:26:44.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.913+0000 7f4517695640 1 -- 172.21.15.123:0/2379344112 wait complete. 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.914+0000 7f4517695640 10 monclient: build_initial_monmap 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.914+0000 7f4517695640 10 monclient: monmap: 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.914+0000 7f4517695640 1 librados: starting msgr at 2024-03-25T19:26:44.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.914+0000 7f4517695640 1 librados: starting objecter 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.914+0000 7f4517695640 1 Processor -- start 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.914+0000 7f4517695640 1 -- start start 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.914+0000 7f4517695640 1 librados: setting wanted keys 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.914+0000 7f4517695640 1 librados: calling monclient init 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.914+0000 7f4517695640 10 monclient: init 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.915+0000 7f4517695640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.915+0000 7f4517695640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.915+0000 7f4517695640 10 monclient(hunting): picked mon.noname-b con 0x7f45100a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.915+0000 7f4517695640 10 monclient(hunting): picked mon.noname-c con 0x7f451013f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.915+0000 7f4517695640 10 monclient(hunting): picked mon.noname-a con 0x7f45101430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.915+0000 7f4517695640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510138980 con 0x7f45101430c0 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.915+0000 7f4517695640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510138640 con 0x7f45100a98f0 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.915+0000 7f4517695640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100b1700 con 0x7f451013f710 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.915+0000 7f4517695640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:44.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.915+0000 7f4517695640 10 monclient(hunting): authenticate will time out at 5842.276129s 2024-03-25T19:26:44.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.926+0000 7f4515c32640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f45101430c0 0x7f45101db000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:40852/0 (socket says 172.21.15.123:40852) 2024-03-25T19:26:44.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.926+0000 7f4515c32640 1 -- 172.21.15.123:0/2309572661 learned_addr learned my addr 172.21.15.123:0/2309572661 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:44.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.927+0000 7f4505ffb640 10 client.?.objecter ms_handle_connect 0x7f45101430c0 2024-03-25T19:26:44.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.927+0000 7f4505ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:44.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.938+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067777032 0 0) 0x7f4510138980 con 0x7f45101430c0 2024-03-25T19:26:44.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.938+0000 7f4505ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:44.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.938+0000 7f4505ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:44.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.938+0000 7f4505ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:44.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.938+0000 7f4505ffb640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:26:44.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.938+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f4004eb0 con 0x7f45101430c0 2024-03-25T19:26:44.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.949+0000 7f4505ffb640 10 client.?.objecter ms_handle_connect 0x7f45100a98f0 2024-03-25T19:26:44.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.949+0000 7f4505ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:44.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.949+0000 7f4505ffb640 10 client.?.objecter ms_handle_connect 0x7f451013f710 2024-03-25T19:26:44.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.949+0000 7f4505ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:44.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.953+0000 7f4515431640 1 -- 172.21.15.123:0/2309572661 >> v1:172.21.15.151:6789/0 conn(0x7f45100a98f0 legacy=0x7f4510137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:26:44.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.953+0000 7f4515431640 1 -- 172.21.15.123:0/2309572661 >> v1:172.21.15.151:6789/0 conn(0x7f45100a98f0 legacy=0x7f4510137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:26:44.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.953+0000 7f4515431640 1 --1- 172.21.15.123:0/2309572661 >> v1:172.21.15.151:6789/0 conn(0x7f45100a98f0 0x7f4510137960 :-1 s=OPENED pgs=28695 cs=1 l=1).handle_message read tag failed 2024-03-25T19:26:44.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.953+0000 7f4515431640 1 --1- 172.21.15.123:0/2309572661 >> v1:172.21.15.151:6789/0 conn(0x7f45100a98f0 0x7f4510137960 :-1 s=OPENED pgs=28695 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:26:44.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.953+0000 7f4505ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.151:6789/0 2024-03-25T19:26:44.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.961+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554477578 0 0) 0x7f44f4004eb0 con 0x7f45101430c0 2024-03-25T19:26:44.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.961+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4510138980 con 0x7f45101430c0 2024-03-25T19:26:44.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.961+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4508001790 con 0x7f45101430c0 2024-03-25T19:26:44.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.961+0000 7f4505ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:44.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.961+0000 7f4505ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:44.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.961+0000 7f4505ffb640 10 monclient(hunting): dump: 2024-03-25T19:26:44.962 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:44.962 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:44.963 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:44.963 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:44.963 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:44.963 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:44.963 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:44.963 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:44.963 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:44.963 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:44.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:44.961+0000 7f4505ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 607442181 0 0) 0x7f4510138980 con 0x7f45101430c0 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 >> v1:172.21.15.123:6790/0 conn(0x7f451013f710 legacy=0x7f45101da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 >> v1:172.21.15.151:6789/0 conn(0x7f45100a98f0 legacy=0x7f4510137960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 1 monclient: found mon.a 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45100b1290 con 0x7f45101430c0 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 10 monclient: _finish_auth 0 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4505ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4517695640 5 monclient: authenticate success, global_id 4401 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4517695640 10 monclient: _renew_subs 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4517695640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:45.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4517695640 1 -- 172.21.15.123:0/2309572661 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45101e2eb0 con 0x7f45101430c0 2024-03-25T19:26:45.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4517695640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:45.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4517695640 10 monclient: _renew_subs 2024-03-25T19:26:45.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4517695640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:45.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.025+0000 7f4517695640 1 -- 172.21.15.123:0/2309572661 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45101e3220 con 0x7f45101430c0 2024-03-25T19:26:45.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.026+0000 7f4517695640 1 librados: init done 2024-03-25T19:26:45.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.027+0000 7f4517695640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:45.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.027+0000 7f4517695640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:45.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.027+0000 7f4517695640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:45.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.027+0000 7f4517695640 1 -- 172.21.15.123:0/2309572661 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45100abb30 con 0x7f45101430c0 2024-03-25T19:26:45.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.045+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4508001c80 con 0x7f45101430c0 2024-03-25T19:26:45.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.045+0000 7f4505ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:45.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.045+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4508004980 con 0x7f45101430c0 2024-03-25T19:26:45.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.045+0000 7f4505ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:45.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.045+0000 7f4505ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.045+0000 7f4505ffb640 10 monclient: dump: 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:45.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.045+0000 7f45077fe640 4 set_mon_vals no callback set 2024-03-25T19:26:45.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.380+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f4508004ae0 con 0x7f45101430c0 2024-03-25T19:26:45.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.381+0000 7f4515431640 10 monclient: get_auth_request con 0x7f44f406dcc0 auth_method 0 2024-03-25T19:26:45.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.381+0000 7f4505ffb640 10 client.4401.objecter ms_handle_connect 0x7f44f406dcc0 2024-03-25T19:26:45.387 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.386+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f45080546b0 con 0x7f45101430c0 2024-03-25T19:26:45.387 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.386+0000 7f4505ffb640 10 client.4401.objecter ms_dispatch 0x7f4510000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:45.387 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.386+0000 7f4505ffb640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:45.387 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.386+0000 7f4505ffb640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:45.387 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.386+0000 7f4505ffb640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-25T19:26:45.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.386+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f450804fff0 con 0x7f45101430c0 2024-03-25T19:26:45.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.386+0000 7f4505ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:45.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.386+0000 7f4505ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:45.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.642+0000 7f4517695640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:45.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.642+0000 7f4517695640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:45.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.642+0000 7f4517695640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:26:45.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.642+0000 7f4517695640 1 -- 172.21.15.123:0/2309572661 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45100627b0 con 0x7f45101430c0 2024-03-25T19:26:45.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.668+0000 7f4505ffb640 1 -- 172.21.15.123:0/2309572661 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f45080520f0 con 0x7f45101430c0 2024-03-25T19:26:45.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.668+0000 7f4505ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:45.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.668+0000 7f4505ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:45.669 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:26:45.669 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:45.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.670+0000 7f4517695640 10 librados: watch_flush enter 2024-03-25T19:26:45.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 10 librados: watch_flush exit 2024-03-25T19:26:45.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-25T19:26:45.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 10 client.4401.objecter successfully canceled tick 2024-03-25T19:26:45.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 1 -- 172.21.15.123:0/2309572661 >> v1:172.21.15.151:6800/727684239 conn(0x7f44f406dcc0 legacy=0x7f44f4070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:45.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 10 monclient: shutdown 2024-03-25T19:26:45.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:45.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 1 -- 172.21.15.123:0/2309572661 >> v1:172.21.15.123:6789/0 conn(0x7f45101430c0 legacy=0x7f45101db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:45.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 1 -- 172.21.15.123:0/2309572661 shutdown_connections 2024-03-25T19:26:45.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 1 -- 172.21.15.123:0/2309572661 >> 172.21.15.123:0/2309572661 conn(0x7f45100b7ea0 msgr2=0x7f4510074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:45.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 1 -- 172.21.15.123:0/2309572661 shutdown_connections 2024-03-25T19:26:45.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 1 -- 172.21.15.123:0/2309572661 wait complete. 2024-03-25T19:26:45.672 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:45.671+0000 7f4517695640 1 librados: shutdown 2024-03-25T19:26:45.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:50.472 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:04:40.044 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:50.483 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:04:40.055 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:50.519 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:04:40.091 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:50.544 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:04:40.077 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:50.544 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:04:40.116 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:50.551 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:04:40.124 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:50.561 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:04:40.095 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:50.586 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:04:40.158 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:50.588 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:04:40.121 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:52.487 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:52.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.603+0000 7fc627b67640 10 monclient: get_monmap_and_config 2024-03-25T19:26:52.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.604+0000 7fc627b67640 10 monclient: build_initial_monmap 2024-03-25T19:26:52.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.604+0000 7fc627b67640 10 monclient: monmap: 2024-03-25T19:26:52.604 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:52.604 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:52.604 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:26:52.604933+0000 2024-03-25T19:26:52.604 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:26:52.604933+0000 2024-03-25T19:26:52.604 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:52.605 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:52.605 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:52.605 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:26:52.605 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:26:52.605 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:52.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.604+0000 7fc627b67640 1 Processor -- start 2024-03-25T19:26:52.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.604+0000 7fc627b67640 1 -- start start 2024-03-25T19:26:52.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.604+0000 7fc627b67640 10 monclient: init 2024-03-25T19:26:52.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.605+0000 7fc627b67640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:52.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.605+0000 7fc627b67640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:52.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.605+0000 7fc627b67640 10 monclient(hunting): picked mon.noname-b con 0x7fc6200a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:52.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.605+0000 7fc627b67640 10 monclient(hunting): picked mon.noname-c con 0x7fc62013f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:52.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.605+0000 7fc627b67640 10 monclient(hunting): picked mon.noname-a con 0x7fc6201430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:52.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.605+0000 7fc627b67640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc620142520 con 0x7fc6201430c0 2024-03-25T19:26:52.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.605+0000 7fc627b67640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6200b28a0 con 0x7fc62013f710 2024-03-25T19:26:52.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.605+0000 7fc627b67640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6200b2a30 con 0x7fc6200a98f0 2024-03-25T19:26:52.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.605+0000 7fc627b67640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:52.607 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.605+0000 7fc627b67640 10 monclient(hunting): authenticate will time out at 5849.965598s 2024-03-25T19:26:52.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.610+0000 7fc625903640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fc6200a98f0 0x7fc6200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:33362/0 (socket says 172.21.15.123:33362) 2024-03-25T19:26:52.610 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.610+0000 7fc625903640 1 -- 172.21.15.123:0/989253997 learned_addr learned my addr 172.21.15.123:0/989253997 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:52.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.634+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725872170 0 0) 0x7fc6200b2a30 con 0x7fc6200a98f0 2024-03-25T19:26:52.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.634+0000 7fc624901640 10 monclient(hunting): _negotiate 2024-03-25T19:26:52.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.634+0000 7fc624901640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:52.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.634+0000 7fc624901640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:52.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.634+0000 7fc624901640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:26:52.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.634+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc60c003520 con 0x7fc6200a98f0 2024-03-25T19:26:52.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.648+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647509893 0 0) 0x7fc620142520 con 0x7fc6201430c0 2024-03-25T19:26:52.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.648+0000 7fc624901640 10 monclient(hunting): _negotiate 2024-03-25T19:26:52.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.648+0000 7fc624901640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:52.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.648+0000 7fc624901640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:52.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.648+0000 7fc624901640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:26:52.649 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.648+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6200b2a30 con 0x7fc6201430c0 2024-03-25T19:26:52.722 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1307315092 0 0) 0x7fc6200b2a30 con 0x7fc6201430c0 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 >> v1:172.21.15.123:6790/0 conn(0x7fc6200a98f0 legacy=0x7fc6200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 >> v1:172.21.15.151:6789/0 conn(0x7fc62013f710 legacy=0x7fc6201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 10 monclient: _finish_hunting 0 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 1 monclient: found mon.noname-a 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6200b44c0 con 0x7fc6201430c0 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 10 monclient: _finish_auth 0 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 10 monclient: _check_auth_tickets 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc627b67640 5 monclient: authenticate success, global_id 4404 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc6180019f0 con 0x7fc6201430c0 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:52.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc624901640 10 monclient: dump: 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:52.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.722+0000 7fc627b67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:52.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.746+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc618001b50 con 0x7fc6201430c0 2024-03-25T19:26:52.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.746+0000 7fc624901640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:52.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.746+0000 7fc624901640 1 -- 172.21.15.123:0/989253997 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc618001f20 con 0x7fc6201430c0 2024-03-25T19:26:52.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.746+0000 7fc624901640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:52.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.746+0000 7fc624901640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:52.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.746+0000 7fc624901640 10 monclient: dump: 2024-03-25T19:26:52.746 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:52.746 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:52.746 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:52.747 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:52.747 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:52.747 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:52.747 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:52.747 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:52.747 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:52.747 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:52.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.746+0000 7fc627b67640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:52.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.746+0000 7fc627b67640 4 set_mon_vals no callback set 2024-03-25T19:26:52.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.747+0000 7fc627b67640 10 monclient: shutdown 2024-03-25T19:26:52.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.747+0000 7fc627b67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:52.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.747+0000 7fc627b67640 1 -- 172.21.15.123:0/989253997 >> v1:172.21.15.123:6789/0 conn(0x7fc6201430c0 legacy=0x7fc62014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.747+0000 7fc627b67640 1 -- 172.21.15.123:0/989253997 shutdown_connections 2024-03-25T19:26:52.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.747+0000 7fc627b67640 1 -- 172.21.15.123:0/989253997 >> 172.21.15.123:0/989253997 conn(0x7fc6200b7ea0 msgr2=0x7fc6200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.747+0000 7fc627b67640 1 -- 172.21.15.123:0/989253997 shutdown_connections 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.747+0000 7fc627b67640 1 -- 172.21.15.123:0/989253997 wait complete. 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.748+0000 7fc627b67640 10 monclient: build_initial_monmap 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.748+0000 7fc627b67640 10 monclient: monmap: 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:26:52.750 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.748+0000 7fc627b67640 1 librados: starting msgr at 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.748+0000 7fc627b67640 1 librados: starting objecter 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.748+0000 7fc627b67640 1 Processor -- start 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.748+0000 7fc627b67640 1 -- start start 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.748+0000 7fc627b67640 1 librados: setting wanted keys 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.748+0000 7fc627b67640 1 librados: calling monclient init 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.748+0000 7fc627b67640 10 monclient: init 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.749+0000 7fc627b67640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.749+0000 7fc627b67640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.749+0000 7fc627b67640 10 monclient(hunting): picked mon.noname-b con 0x7fc6200a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.749+0000 7fc627b67640 10 monclient(hunting): picked mon.noname-a con 0x7fc62013f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.749+0000 7fc627b67640 10 monclient(hunting): picked mon.noname-c con 0x7fc6201430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.749+0000 7fc627b67640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc620139890 con 0x7fc62013f710 2024-03-25T19:26:52.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.749+0000 7fc627b67640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6201dba10 con 0x7fc6200a98f0 2024-03-25T19:26:52.752 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.749+0000 7fc627b67640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62013ace0 con 0x7fc6201430c0 2024-03-25T19:26:52.752 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.749+0000 7fc627b67640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:52.752 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.749+0000 7fc627b67640 10 monclient(hunting): authenticate will time out at 5850.109657s 2024-03-25T19:26:52.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.770+0000 7fc625102640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fc62013f710 0x7fc6201dabf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:60982/0 (socket says 172.21.15.123:60982) 2024-03-25T19:26:52.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.770+0000 7fc625102640 1 -- 172.21.15.123:0/1293381097 learned_addr learned my addr 172.21.15.123:0/1293381097 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:26:52.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.771+0000 7fc6167fc640 10 client.?.objecter ms_handle_connect 0x7fc62013f710 2024-03-25T19:26:52.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.771+0000 7fc6167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:52.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.793+0000 7fc6167fc640 10 client.?.objecter ms_handle_connect 0x7fc6200a98f0 2024-03-25T19:26:52.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.793+0000 7fc6167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:52.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.797+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866384812 0 0) 0x7fc620139890 con 0x7fc62013f710 2024-03-25T19:26:52.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.797+0000 7fc6167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:52.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.797+0000 7fc6167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:52.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.797+0000 7fc6167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:52.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.797+0000 7fc6167fc640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:26:52.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.797+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc604004eb0 con 0x7fc62013f710 2024-03-25T19:26:52.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.797+0000 7fc6167fc640 10 client.?.objecter ms_handle_connect 0x7fc6201430c0 2024-03-25T19:26:52.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.797+0000 7fc6167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:52.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.801+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973411766 0 0) 0x7fc6201dba10 con 0x7fc6200a98f0 2024-03-25T19:26:52.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.801+0000 7fc6167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:52.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.801+0000 7fc6167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:52.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.801+0000 7fc6167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:52.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.801+0000 7fc6167fc640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:26:52.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.801+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc620139890 con 0x7fc6200a98f0 2024-03-25T19:26:52.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.807+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2568980136 0 0) 0x7fc620139890 con 0x7fc6200a98f0 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.808+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6201dba10 con 0x7fc6200a98f0 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.808+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc61c001950 con 0x7fc6200a98f0 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.808+0000 7fc6167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.808+0000 7fc6167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.808+0000 7fc6167fc640 10 monclient(hunting): dump: 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:52.808 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:52.809 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:52.809 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.808+0000 7fc6167fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:52.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.812+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1567747056 0 0) 0x7fc604004eb0 con 0x7fc62013f710 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.812+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc620139890 con 0x7fc62013f710 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.812+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc610001a30 con 0x7fc62013f710 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.813+0000 7fc6167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.813+0000 7fc6167fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.813+0000 7fc6167fc640 10 monclient(hunting): dump: 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:52.813 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:52.814 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:52.814 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 93795130 0 0) 0x7fc6201dba10 con 0x7fc6200a98f0 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 >> v1:172.21.15.123:6790/0 conn(0x7fc6201430c0 legacy=0x7fc6201db300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 >> v1:172.21.15.123:6789/0 conn(0x7fc62013f710 legacy=0x7fc6201dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 10 monclient: _finish_hunting 0 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 1 monclient: found mon.b 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62013afc0 con 0x7fc6200a98f0 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 10 monclient: _finish_auth 0 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 10 monclient: _check_auth_tickets 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc6167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:52.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc627b67640 5 monclient: authenticate success, global_id 4411 2024-03-25T19:26:52.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc627b67640 10 monclient: _renew_subs 2024-03-25T19:26:52.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc627b67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:52.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc627b67640 1 -- 172.21.15.123:0/1293381097 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62013b2a0 con 0x7fc6200a98f0 2024-03-25T19:26:52.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc627b67640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:52.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc627b67640 10 monclient: _renew_subs 2024-03-25T19:26:52.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc627b67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:52.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc627b67640 1 -- 172.21.15.123:0/1293381097 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6200b7b50 con 0x7fc6200a98f0 2024-03-25T19:26:52.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.817+0000 7fc627b67640 1 librados: init done 2024-03-25T19:26:52.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.819+0000 7fc627b67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:52.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.819+0000 7fc627b67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:52.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.819+0000 7fc627b67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:52.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:52.819+0000 7fc627b67640 1 -- 172.21.15.123:0/1293381097 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc62014f320 con 0x7fc6200a98f0 2024-03-25T19:26:53.094 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.094+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc61c001d60 con 0x7fc6200a98f0 2024-03-25T19:26:53.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.094+0000 7fc6167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:53.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.094+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc61c004980 con 0x7fc6200a98f0 2024-03-25T19:26:53.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.094+0000 7fc6167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:53.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.094+0000 7fc6167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:53.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.094+0000 7fc617fff640 4 set_mon_vals no callback set 2024-03-25T19:26:53.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.094+0000 7fc6167fc640 10 monclient: dump: 2024-03-25T19:26:53.095 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:26:53.095 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:26:53.095 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:53.095 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:26:53.096 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:53.096 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:26:53.096 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:26:53.096 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:26:53.096 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:26:53.096 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:26:53.096 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.095+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fc61c004ba0 con 0x7fc6200a98f0 2024-03-25T19:26:53.096 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.096+0000 7fc625102640 10 monclient: get_auth_request con 0x7fc60406ddf0 auth_method 0 2024-03-25T19:26:53.097 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.096+0000 7fc6167fc640 10 client.4411.objecter ms_handle_connect 0x7fc60406ddf0 2024-03-25T19:26:53.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.100+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fc61c054740 con 0x7fc6200a98f0 2024-03-25T19:26:53.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.100+0000 7fc6167fc640 10 client.4411.objecter ms_dispatch 0x7fc620000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:53.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.100+0000 7fc6167fc640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:53.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.100+0000 7fc6167fc640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:53.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.100+0000 7fc6167fc640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T19:26:53.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.100+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc61c050080 con 0x7fc6200a98f0 2024-03-25T19:26:53.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.100+0000 7fc6167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:53.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.100+0000 7fc6167fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:53.352 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.352+0000 7fc627b67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:53.353 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.352+0000 7fc627b67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:53.353 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.352+0000 7fc627b67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:26:53.353 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.352+0000 7fc627b67640 1 -- 172.21.15.123:0/1293381097 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6200627b0 con 0x7fc6200a98f0 2024-03-25T19:26:53.372 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.372+0000 7fc6167fc640 1 -- 172.21.15.123:0/1293381097 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fc61c052180 con 0x7fc6200a98f0 2024-03-25T19:26:53.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.372+0000 7fc6167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:53.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.372+0000 7fc6167fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:53.373 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:26:53.373 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:53.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.374+0000 7fc627b67640 10 librados: watch_flush enter 2024-03-25T19:26:53.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.374+0000 7fc627b67640 10 librados: watch_flush exit 2024-03-25T19:26:53.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.374+0000 7fc627b67640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T19:26:53.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.374+0000 7fc627b67640 10 client.4411.objecter successfully canceled tick 2024-03-25T19:26:53.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.375+0000 7fc627b67640 1 -- 172.21.15.123:0/1293381097 >> v1:172.21.15.151:6800/727684239 conn(0x7fc60406ddf0 legacy=0x7fc6040702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:53.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.375+0000 7fc627b67640 10 monclient: shutdown 2024-03-25T19:26:53.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.375+0000 7fc627b67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:53.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.375+0000 7fc627b67640 1 -- 172.21.15.123:0/1293381097 >> v1:172.21.15.151:6789/0 conn(0x7fc6200a98f0 legacy=0x7fc620139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:53.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.375+0000 7fc627b67640 1 -- 172.21.15.123:0/1293381097 shutdown_connections 2024-03-25T19:26:53.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.375+0000 7fc627b67640 1 -- 172.21.15.123:0/1293381097 >> 172.21.15.123:0/1293381097 conn(0x7fc6200b7ea0 msgr2=0x7fc620074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:53.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.375+0000 7fc627b67640 1 -- 172.21.15.123:0/1293381097 shutdown_connections 2024-03-25T19:26:53.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.375+0000 7fc627b67640 1 -- 172.21.15.123:0/1293381097 wait complete. 2024-03-25T19:26:53.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:26:53.375+0000 7fc627b67640 1 librados: shutdown 2024-03-25T19:26:53.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:00.194 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:00.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.309+0000 7fab57bc6640 10 monclient: get_monmap_and_config 2024-03-25T19:27:00.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.309+0000 7fab57bc6640 10 monclient: build_initial_monmap 2024-03-25T19:27:00.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.309+0000 7fab57bc6640 10 monclient: monmap: 2024-03-25T19:27:00.309 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:00.309 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:00.310 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:27:00.310283+0000 2024-03-25T19:27:00.310 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:27:00.310283+0000 2024-03-25T19:27:00.310 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:00.310 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:00.310 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:00.310 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:27:00.310 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:27:00.310 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:00.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.310+0000 7fab57bc6640 1 Processor -- start 2024-03-25T19:27:00.311 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.310+0000 7fab57bc6640 1 -- start start 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.310+0000 7fab57bc6640 10 monclient: init 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.310+0000 7fab57bc6640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.310+0000 7fab57bc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.310+0000 7fab57bc6640 10 monclient(hunting): picked mon.noname-c con 0x7fab500a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.311+0000 7fab57bc6640 10 monclient(hunting): picked mon.noname-a con 0x7fab5013f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.311+0000 7fab57bc6640 10 monclient(hunting): picked mon.noname-b con 0x7fab501430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.311+0000 7fab57bc6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab50142520 con 0x7fab5013f710 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.311+0000 7fab57bc6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500b28a0 con 0x7fab500a98f0 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.311+0000 7fab57bc6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500b2a30 con 0x7fab501430c0 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.311+0000 7fab57bc6640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:00.312 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.311+0000 7fab57bc6640 10 monclient(hunting): authenticate will time out at 5857.670959s 2024-03-25T19:27:00.321 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.321+0000 7fab55962640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fab500a98f0 0x7fab500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:49390/0 (socket says 172.21.15.123:49390) 2024-03-25T19:27:00.321 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.321+0000 7fab55962640 1 -- 172.21.15.123:0/297055602 learned_addr learned my addr 172.21.15.123:0/297055602 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:00.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836142028 0 0) 0x7fab500b2a30 con 0x7fab501430c0 2024-03-25T19:27:00.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 10 monclient(hunting): _negotiate 2024-03-25T19:27:00.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:00.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:00.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:27:00.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab3c003520 con 0x7fab501430c0 2024-03-25T19:27:00.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659235811 0 0) 0x7fab500b28a0 con 0x7fab500a98f0 2024-03-25T19:27:00.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 10 monclient(hunting): _negotiate 2024-03-25T19:27:00.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:00.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:00.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:27:00.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.348+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab500b2a30 con 0x7fab500a98f0 2024-03-25T19:27:00.359 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.359+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716650513 0 0) 0x7fab50142520 con 0x7fab5013f710 2024-03-25T19:27:00.359 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.359+0000 7fab54960640 10 monclient(hunting): _negotiate 2024-03-25T19:27:00.359 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.359+0000 7fab54960640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:00.359 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.359+0000 7fab54960640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:00.359 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.359+0000 7fab54960640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:27:00.359 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.359+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab500b28a0 con 0x7fab5013f710 2024-03-25T19:27:00.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2971793658 0 0) 0x7fab500b2a30 con 0x7fab500a98f0 2024-03-25T19:27:00.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 >> v1:172.21.15.123:6790/0 conn(0x7fab501430c0 legacy=0x7fab5014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:00.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 >> v1:172.21.15.123:6789/0 conn(0x7fab5013f710 legacy=0x7fab501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:00.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 10 monclient: _finish_hunting 0 2024-03-25T19:27:00.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 1 monclient: found mon.noname-c 2024-03-25T19:27:00.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:00.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500b44c0 con 0x7fab500a98f0 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 10 monclient: _finish_auth 0 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 10 monclient: _check_auth_tickets 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fab4c001670 con 0x7fab500a98f0 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab54960640 10 monclient: dump: 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:00.363 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:00.364 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:00.364 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:00.364 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:00.365 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:00.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab57bc6640 5 monclient: authenticate success, global_id 4414 2024-03-25T19:27:00.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.362+0000 7fab57bc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:00.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.379+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab4c0017d0 con 0x7fab500a98f0 2024-03-25T19:27:00.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.379+0000 7fab54960640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:00.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.379+0000 7fab54960640 1 -- 172.21.15.123:0/297055602 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fab4c001be0 con 0x7fab500a98f0 2024-03-25T19:27:00.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.379+0000 7fab57bc6640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:00.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.379+0000 7fab57bc6640 4 set_mon_vals no callback set 2024-03-25T19:27:00.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.379+0000 7fab54960640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:00.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.379+0000 7fab54960640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:00.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.379+0000 7fab54960640 10 monclient: dump: 2024-03-25T19:27:00.379 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:00.379 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:00.380 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:00.380 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:00.380 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:00.380 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:00.380 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:00.380 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:00.380 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:00.380 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:00.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.380+0000 7fab57bc6640 10 monclient: shutdown 2024-03-25T19:27:00.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.380+0000 7fab57bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:00.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.380+0000 7fab57bc6640 1 -- 172.21.15.123:0/297055602 >> v1:172.21.15.151:6789/0 conn(0x7fab500a98f0 legacy=0x7fab500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.380+0000 7fab57bc6640 1 -- 172.21.15.123:0/297055602 shutdown_connections 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.380+0000 7fab57bc6640 1 -- 172.21.15.123:0/297055602 >> 172.21.15.123:0/297055602 conn(0x7fab500b7ea0 msgr2=0x7fab500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.380+0000 7fab57bc6640 1 -- 172.21.15.123:0/297055602 shutdown_connections 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.380+0000 7fab57bc6640 1 -- 172.21.15.123:0/297055602 wait complete. 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.381+0000 7fab57bc6640 10 monclient: build_initial_monmap 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.381+0000 7fab57bc6640 10 monclient: monmap: 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:27:00.382 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:00.383 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:27:00.383 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:27:00.383 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:00.383 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.381+0000 7fab57bc6640 1 librados: starting msgr at 2024-03-25T19:27:00.383 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.381+0000 7fab57bc6640 1 librados: starting objecter 2024-03-25T19:27:00.383 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.381+0000 7fab57bc6640 1 Processor -- start 2024-03-25T19:27:00.383 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.381+0000 7fab57bc6640 1 -- start start 2024-03-25T19:27:00.383 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.381+0000 7fab57bc6640 1 librados: setting wanted keys 2024-03-25T19:27:00.383 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.381+0000 7fab57bc6640 1 librados: calling monclient init 2024-03-25T19:27:00.383 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.381+0000 7fab57bc6640 10 monclient: init 2024-03-25T19:27:00.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.382+0000 7fab57bc6640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:00.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.382+0000 7fab57bc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:00.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.382+0000 7fab57bc6640 10 monclient(hunting): picked mon.noname-c con 0x7fab500a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:00.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.382+0000 7fab57bc6640 10 monclient(hunting): picked mon.noname-a con 0x7fab5013f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:00.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.382+0000 7fab57bc6640 10 monclient(hunting): picked mon.noname-b con 0x7fab501430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:00.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.382+0000 7fab57bc6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab50139330 con 0x7fab5013f710 2024-03-25T19:27:00.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.382+0000 7fab57bc6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500abe70 con 0x7fab501430c0 2024-03-25T19:27:00.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.382+0000 7fab57bc6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500ac150 con 0x7fab500a98f0 2024-03-25T19:27:00.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.382+0000 7fab57bc6640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:00.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.382+0000 7fab57bc6640 10 monclient(hunting): authenticate will time out at 5857.742685s 2024-03-25T19:27:00.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.390+0000 7fab55962640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fab500a98f0 0x7fab501389d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:37220/0 (socket says 172.21.15.123:37220) 2024-03-25T19:27:00.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.390+0000 7fab55962640 1 -- 172.21.15.123:0/2273329758 learned_addr learned my addr 172.21.15.123:0/2273329758 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:00.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.391+0000 7fab467fc640 10 client.?.objecter ms_handle_connect 0x7fab500a98f0 2024-03-25T19:27:00.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.391+0000 7fab467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:00.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.394+0000 7fab467fc640 10 client.?.objecter ms_handle_connect 0x7fab501430c0 2024-03-25T19:27:00.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.394+0000 7fab467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:00.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.398+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565600983 0 0) 0x7fab500ac150 con 0x7fab500a98f0 2024-03-25T19:27:00.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.398+0000 7fab467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:00.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.398+0000 7fab467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:00.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.398+0000 7fab467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:00.399 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.398+0000 7fab467fc640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:27:00.399 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.398+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab34004eb0 con 0x7fab500a98f0 2024-03-25T19:27:00.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.406+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 610333388 0 0) 0x7fab34004eb0 con 0x7fab500a98f0 2024-03-25T19:27:00.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.406+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab500ac150 con 0x7fab500a98f0 2024-03-25T19:27:00.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.406+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fab4c001c20 con 0x7fab500a98f0 2024-03-25T19:27:00.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.406+0000 7fab467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:00.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.406+0000 7fab467fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:00.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.406+0000 7fab467fc640 10 monclient(hunting): dump: 2024-03-25T19:27:00.406 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:00.406 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:00.406 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:00.407 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:00.407 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:00.407 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:00.407 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:00.407 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:00.407 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:00.407 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:00.407 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.406+0000 7fab467fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:00.407 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.407+0000 7fab467fc640 10 client.?.objecter ms_handle_connect 0x7fab5013f710 2024-03-25T19:27:00.408 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.407+0000 7fab467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:00.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.410+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695302671 0 0) 0x7fab500abe70 con 0x7fab501430c0 2024-03-25T19:27:00.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.410+0000 7fab467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:00.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.410+0000 7fab467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:00.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.410+0000 7fab467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:00.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.410+0000 7fab467fc640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:27:00.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.410+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab34004eb0 con 0x7fab501430c0 2024-03-25T19:27:00.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2144855118 0 0) 0x7fab500ac150 con 0x7fab500a98f0 2024-03-25T19:27:00.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 >> v1:172.21.15.151:6789/0 conn(0x7fab501430c0 legacy=0x7fab501e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:00.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 >> v1:172.21.15.123:6789/0 conn(0x7fab5013f710 legacy=0x7fab500ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:00.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:00.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 1 monclient: found mon.c 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500acc80 con 0x7fab500a98f0 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab57bc6640 5 monclient: authenticate success, global_id 4436 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab57bc6640 10 monclient: _renew_subs 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab57bc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab57bc6640 1 -- 172.21.15.123:0/2273329758 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab50137e30 con 0x7fab500a98f0 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab57bc6640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab57bc6640 10 monclient: _renew_subs 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab57bc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:27:00.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab57bc6640 1 -- 172.21.15.123:0/2273329758 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab50136880 con 0x7fab500a98f0 2024-03-25T19:27:00.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.414+0000 7fab57bc6640 1 librados: init done 2024-03-25T19:27:00.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.415+0000 7fab57bc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:00.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.415+0000 7fab57bc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:00.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.415+0000 7fab57bc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:27:00.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.415+0000 7fab57bc6640 1 -- 172.21.15.123:0/2273329758 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab50136b30 con 0x7fab500a98f0 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.729+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab4c004a90 con 0x7fab500a98f0 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.729+0000 7fab467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.729+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fab4c004e60 con 0x7fab500a98f0 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.729+0000 7fab467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.729+0000 7fab467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.729+0000 7fab467fc640 10 monclient: dump: 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:00.730 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:00.731 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:00.731 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:00.731 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:00.731 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:00.731 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:00.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.729+0000 7fab47fff640 4 set_mon_vals no callback set 2024-03-25T19:27:00.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.730+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fab4c005080 con 0x7fab500a98f0 2024-03-25T19:27:00.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.731+0000 7fab55161640 10 monclient: get_auth_request con 0x7fab3406ddb0 auth_method 0 2024-03-25T19:27:00.732 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.731+0000 7fab467fc640 10 client.4436.objecter ms_handle_connect 0x7fab3406ddb0 2024-03-25T19:27:00.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.738+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fab4c054a20 con 0x7fab500a98f0 2024-03-25T19:27:00.738 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.738+0000 7fab467fc640 10 client.4436.objecter ms_dispatch 0x7fab50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:00.738 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.738+0000 7fab467fc640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:00.738 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.738+0000 7fab467fc640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:00.738 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.738+0000 7fab467fc640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-25T19:27:00.738 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.738+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab4c050530 con 0x7fab500a98f0 2024-03-25T19:27:00.738 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.738+0000 7fab467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:00.738 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.738+0000 7fab467fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:00.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.953+0000 7fab57bc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:00.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.953+0000 7fab57bc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:00.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.953+0000 7fab57bc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:27:00.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:00.953+0000 7fab57bc6640 1 -- 172.21.15.123:0/2273329758 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab500627b0 con 0x7fab500a98f0 2024-03-25T19:27:01.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.105+0000 7fab467fc640 1 -- 172.21.15.123:0/2273329758 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fab4c052460 con 0x7fab500a98f0 2024-03-25T19:27:01.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.106+0000 7fab467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:01.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.106+0000 7fab467fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:01.106 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:27:01.106 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:01.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 10 librados: watch_flush enter 2024-03-25T19:27:01.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 10 librados: watch_flush exit 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 10 client.4436.objecter successfully canceled tick 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 1 -- 172.21.15.123:0/2273329758 >> v1:172.21.15.151:6800/727684239 conn(0x7fab3406ddb0 legacy=0x7fab34070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 10 monclient: shutdown 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 1 -- 172.21.15.123:0/2273329758 >> v1:172.21.15.123:6790/0 conn(0x7fab500a98f0 legacy=0x7fab501389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 1 -- 172.21.15.123:0/2273329758 shutdown_connections 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 1 -- 172.21.15.123:0/2273329758 >> 172.21.15.123:0/2273329758 conn(0x7fab500b7ea0 msgr2=0x7fab50074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.108+0000 7fab57bc6640 1 -- 172.21.15.123:0/2273329758 shutdown_connections 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.109+0000 7fab57bc6640 1 -- 172.21.15.123:0/2273329758 wait complete. 2024-03-25T19:27:01.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:01.109+0000 7fab57bc6640 1 librados: shutdown 2024-03-25T19:27:01.121 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:07.929 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:08.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.045+0000 7f2cb61ab640 10 monclient: get_monmap_and_config 2024-03-25T19:27:08.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.045+0000 7f2cb61ab640 10 monclient: build_initial_monmap 2024-03-25T19:27:08.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.045+0000 7f2cb61ab640 10 monclient: monmap: 2024-03-25T19:27:08.045 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:08.045 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:08.045 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:27:08.046388+0000 2024-03-25T19:27:08.046 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:27:08.046388+0000 2024-03-25T19:27:08.046 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:08.046 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:08.046 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:08.046 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:27:08.046 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:27:08.046 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:08.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.046+0000 7f2cb61ab640 1 Processor -- start 2024-03-25T19:27:08.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.046+0000 7f2cb61ab640 1 -- start start 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.046+0000 7f2cb61ab640 10 monclient: init 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.047+0000 7f2cb61ab640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.047+0000 7f2cb61ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.047+0000 7f2cb61ab640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb00a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.047+0000 7f2cb61ab640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb013f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.047+0000 7f2cb61ab640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb01430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.047+0000 7f2cb61ab640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb0142520 con 0x7f2cb01430c0 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.047+0000 7f2cb61ab640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb00b28a0 con 0x7f2cb00a98f0 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.047+0000 7f2cb61ab640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb00b2a30 con 0x7f2cb013f710 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.047+0000 7f2cb61ab640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:08.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.047+0000 7f2cb61ab640 10 monclient(hunting): authenticate will time out at 5865.407237s 2024-03-25T19:27:08.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.049+0000 7f2caeffd640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f2cb013f710 0x7f2cb01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:37538/0 (socket says 172.21.15.123:37538) 2024-03-25T19:27:08.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.049+0000 7f2caeffd640 1 -- 172.21.15.123:0/2553907586 learned_addr learned my addr 172.21.15.123:0/2553907586 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:08.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.057+0000 7f2cae7fc640 1 -- 172.21.15.123:0/2553907586 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325699035 0 0) 0x7f2cb00b2a30 con 0x7f2cb013f710 2024-03-25T19:27:08.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.057+0000 7f2cae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:08.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.057+0000 7f2cae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:08.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.057+0000 7f2cae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:08.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.057+0000 7f2cae7fc640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:27:08.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.057+0000 7f2cae7fc640 1 -- 172.21.15.123:0/2553907586 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c98003520 con 0x7f2cb013f710 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 1 -- 172.21.15.123:0/2553907586 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1454544268 0 0) 0x7f2c98003520 con 0x7f2cb013f710 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 1 -- 172.21.15.123:0/2553907586 >> v1:172.21.15.151:6789/0 conn(0x7f2cb00a98f0 legacy=0x7f2cb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 1 -- 172.21.15.123:0/2553907586 >> v1:172.21.15.123:6789/0 conn(0x7f2cb01430c0 legacy=0x7f2cb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 1 monclient: found mon.noname-b 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 1 -- 172.21.15.123:0/2553907586 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb00b44c0 con 0x7f2cb013f710 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:08.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 1 -- 172.21.15.123:0/2553907586 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2ca4001690 con 0x7f2cb013f710 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cae7fc640 10 monclient: dump: 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:08.066 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:08.067 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cb61ab640 5 monclient: authenticate success, global_id 4439 2024-03-25T19:27:08.067 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.064+0000 7f2cb61ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:08.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cae7fc640 1 -- 172.21.15.123:0/2553907586 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ca4001860 con 0x7f2cb013f710 2024-03-25T19:27:08.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:08.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cb61ab640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:08.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cb61ab640 4 set_mon_vals no callback set 2024-03-25T19:27:08.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cae7fc640 1 -- 172.21.15.123:0/2553907586 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2ca4001cb0 con 0x7f2cb013f710 2024-03-25T19:27:08.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:08.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:08.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cae7fc640 10 monclient: dump: 2024-03-25T19:27:08.080 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:08.080 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cb61ab640 10 monclient: shutdown 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cb61ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2553907586 >> v1:172.21.15.123:6790/0 conn(0x7f2cb013f710 legacy=0x7f2cb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:08.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2553907586 shutdown_connections 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2553907586 >> 172.21.15.123:0/2553907586 conn(0x7f2cb00b7ea0 msgr2=0x7f2cb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2553907586 shutdown_connections 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.078+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2553907586 wait complete. 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.079+0000 7f2cb61ab640 10 monclient: build_initial_monmap 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.079+0000 7f2cb61ab640 10 monclient: monmap: 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:27:08.083 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:08.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.080+0000 7f2cb61ab640 1 librados: starting msgr at 2024-03-25T19:27:08.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.080+0000 7f2cb61ab640 1 librados: starting objecter 2024-03-25T19:27:08.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.080+0000 7f2cb61ab640 1 Processor -- start 2024-03-25T19:27:08.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.080+0000 7f2cb61ab640 1 -- start start 2024-03-25T19:27:08.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.080+0000 7f2cb61ab640 1 librados: setting wanted keys 2024-03-25T19:27:08.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.080+0000 7f2cb61ab640 1 librados: calling monclient init 2024-03-25T19:27:08.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.080+0000 7f2cb61ab640 10 monclient: init 2024-03-25T19:27:08.084 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.081+0000 7f2cb61ab640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:08.084 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.081+0000 7f2cb61ab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:08.084 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.081+0000 7f2cb61ab640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb00a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.081+0000 7f2cb61ab640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb013f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.081+0000 7f2cb61ab640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb01430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.081+0000 7f2cb61ab640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb01da8f0 con 0x7f2cb00a98f0 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.081+0000 7f2cb61ab640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb01db770 con 0x7f2cb013f710 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.081+0000 7f2cb61ab640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb01dab40 con 0x7f2cb01430c0 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.081+0000 7f2cb61ab640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.081+0000 7f2cb61ab640 10 monclient(hunting): authenticate will time out at 5865.441469s 2024-03-25T19:27:08.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.090+0000 7f2caeffd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f2cb013f710 0x7f2cb0138210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:56428/0 (socket says 172.21.15.123:56428) 2024-03-25T19:27:08.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.090+0000 7f2caf7fe640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2cb00a98f0 0x7f2cb01db060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:40594/0 (socket says 172.21.15.123:40594) 2024-03-25T19:27:08.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.090+0000 7f2caeffd640 1 -- 172.21.15.123:0/2285375705 learned_addr learned my addr 172.21.15.123:0/2285375705 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:08.091 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.091+0000 7f2c97fff640 10 client.?.objecter ms_handle_connect 0x7f2cb013f710 2024-03-25T19:27:08.091 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.091+0000 7f2c97fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:08.091 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.091+0000 7f2c97fff640 10 client.?.objecter ms_handle_connect 0x7f2cb00a98f0 2024-03-25T19:27:08.091 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.091+0000 7f2c97fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:08.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.098+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301676453 0 0) 0x7f2cb01db770 con 0x7f2cb013f710 2024-03-25T19:27:08.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.098+0000 7f2c97fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:08.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.098+0000 7f2c97fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:08.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.098+0000 7f2c97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:08.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.098+0000 7f2c97fff640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:27:08.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.098+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c9c004eb0 con 0x7f2cb013f710 2024-03-25T19:27:08.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.101+0000 7f2c97fff640 10 client.?.objecter ms_handle_connect 0x7f2cb01430c0 2024-03-25T19:27:08.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.101+0000 7f2c97fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:08.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.104+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184946615 0 0) 0x7f2cb01da8f0 con 0x7f2cb00a98f0 2024-03-25T19:27:08.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.104+0000 7f2c97fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:08.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.104+0000 7f2c97fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:08.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.104+0000 7f2c97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:08.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.104+0000 7f2c97fff640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:27:08.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.104+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb01db770 con 0x7f2cb00a98f0 2024-03-25T19:27:08.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049075878 0 0) 0x7f2cb01dab40 con 0x7f2cb01430c0 2024-03-25T19:27:08.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:08.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:08.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:08.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:27:08.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb01da8f0 con 0x7f2cb01430c0 2024-03-25T19:27:08.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2190602889 0 0) 0x7f2c9c004eb0 con 0x7f2cb013f710 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb01dab40 con 0x7f2cb013f710 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2ca4001b70 con 0x7f2cb013f710 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 10 monclient(hunting): dump: 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:08.122 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:08.123 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:08.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.121+0000 7f2c97fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.131+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 597473577 0 0) 0x7f2cb01db770 con 0x7f2cb00a98f0 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.131+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c9c004eb0 con 0x7f2cb00a98f0 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.131+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2ca0001820 con 0x7f2cb00a98f0 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.131+0000 7f2c97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.131+0000 7f2c97fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.131+0000 7f2c97fff640 10 monclient(hunting): dump: 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:08.131 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:08.132 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:08.132 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:08.132 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:08.132 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3279525923 0 0) 0x7f2cb01dab40 con 0x7f2cb013f710 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 >> v1:172.21.15.123:6790/0 conn(0x7f2cb01430c0 legacy=0x7f2cb01e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 >> v1:172.21.15.123:6789/0 conn(0x7f2cb00a98f0 legacy=0x7f2cb01db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 1 monclient: found mon.b 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb0138dc0 con 0x7f2cb013f710 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2c97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:08.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2cb61ab640 5 monclient: authenticate success, global_id 4420 2024-03-25T19:27:08.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2cb61ab640 10 monclient: _renew_subs 2024-03-25T19:27:08.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2cb61ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:08.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.137+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2285375705 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb01378e0 con 0x7f2cb013f710 2024-03-25T19:27:08.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.138+0000 7f2cb61ab640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:08.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.138+0000 7f2cb61ab640 10 monclient: _renew_subs 2024-03-25T19:27:08.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.138+0000 7f2cb61ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:08.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.138+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2285375705 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb0138b00 con 0x7f2cb013f710 2024-03-25T19:27:08.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.138+0000 7f2cb61ab640 1 librados: init done 2024-03-25T19:27:08.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.139+0000 7f2cb61ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:08.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.139+0000 7f2cb61ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:08.140 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.139+0000 7f2cb61ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:08.140 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.139+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2285375705 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb01dacd0 con 0x7f2cb013f710 2024-03-25T19:27:08.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.414+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ca4004bf0 con 0x7f2cb013f710 2024-03-25T19:27:08.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.414+0000 7f2c97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:08.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.414+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f2ca4004f00 con 0x7f2cb013f710 2024-03-25T19:27:08.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.414+0000 7f2c97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:08.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.414+0000 7f2c97fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:08.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.414+0000 7f2c97fff640 10 monclient: dump: 2024-03-25T19:27:08.414 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:08.414 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:08.415 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:08.415 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:08.415 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:08.415 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:08.415 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:08.415 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:08.415 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:08.415 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:08.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.414+0000 7f2cadffb640 4 set_mon_vals no callback set 2024-03-25T19:27:08.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.415+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f2ca401aed0 con 0x7f2cb013f710 2024-03-25T19:27:08.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.415+0000 7f2caf7fe640 10 monclient: get_auth_request con 0x7f2c9c06df10 auth_method 0 2024-03-25T19:27:08.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.416+0000 7f2c97fff640 10 client.4420.objecter ms_handle_connect 0x7f2c9c06df10 2024-03-25T19:27:08.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.420+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f2ca4054a10 con 0x7f2cb013f710 2024-03-25T19:27:08.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.420+0000 7f2c97fff640 10 client.4420.objecter ms_dispatch 0x7f2cb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:08.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.420+0000 7f2c97fff640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:08.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.420+0000 7f2c97fff640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:08.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.420+0000 7f2c97fff640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T19:27:08.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.421+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ca4050520 con 0x7f2cb013f710 2024-03-25T19:27:08.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.421+0000 7f2c97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:08.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.421+0000 7f2c97fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:08.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.672+0000 7f2cb61ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:08.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.672+0000 7f2cb61ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:08.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.672+0000 7f2cb61ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:08.673 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.672+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2285375705 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cb00627b0 con 0x7f2cb013f710 2024-03-25T19:27:08.786 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.786+0000 7f2c97fff640 1 -- 172.21.15.123:0/2285375705 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f2ca4052450 con 0x7f2cb013f710 2024-03-25T19:27:08.786 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.786+0000 7f2c97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:08.786 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.786+0000 7f2c97fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:08.786 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:27:08.787 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:08.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.788+0000 7f2cb61ab640 10 librados: watch_flush enter 2024-03-25T19:27:08.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 10 librados: watch_flush exit 2024-03-25T19:27:08.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T19:27:08.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 10 client.4420.objecter successfully canceled tick 2024-03-25T19:27:08.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2285375705 >> v1:172.21.15.151:6800/727684239 conn(0x7f2c9c06df10 legacy=0x7f2c9c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:08.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 10 monclient: shutdown 2024-03-25T19:27:08.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:08.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2285375705 >> v1:172.21.15.151:6789/0 conn(0x7f2cb013f710 legacy=0x7f2cb0138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:08.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2285375705 shutdown_connections 2024-03-25T19:27:08.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2285375705 >> 172.21.15.123:0/2285375705 conn(0x7f2cb00b7ea0 msgr2=0x7f2cb0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:08.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2285375705 shutdown_connections 2024-03-25T19:27:08.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 1 -- 172.21.15.123:0/2285375705 wait complete. 2024-03-25T19:27:08.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:08.789+0000 7f2cb61ab640 1 librados: shutdown 2024-03-25T19:27:08.801 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:10.467 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:05:00.038 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:10.519 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:05:00.091 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:10.546 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:05:00.118 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:10.565 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:05:00.138 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:10.567 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:05:00.101 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:10.570 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:05:00.103 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:10.581 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:05:00.153 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:10.605 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:05:00.138 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:10.606 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:05:00.178 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:15.608 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:15.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.724+0000 7f8e95bcf640 10 monclient: get_monmap_and_config 2024-03-25T19:27:15.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.724+0000 7f8e95bcf640 10 monclient: build_initial_monmap 2024-03-25T19:27:15.724 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.724+0000 7f8e95bcf640 10 monclient: monmap: 2024-03-25T19:27:15.724 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:15.724 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:15.724 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:27:15.725090+0000 2024-03-25T19:27:15.724 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:27:15.725090+0000 2024-03-25T19:27:15.725 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:15.725 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:15.725 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:15.725 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:27:15.725 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:27:15.725 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:15.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.725+0000 7f8e95bcf640 1 Processor -- start 2024-03-25T19:27:15.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.725+0000 7f8e95bcf640 1 -- start start 2024-03-25T19:27:15.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.725+0000 7f8e95bcf640 10 monclient: init 2024-03-25T19:27:15.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.726+0000 7f8e95bcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:15.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.726+0000 7f8e95bcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:15.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.726+0000 7f8e95bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f8e900705b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:15.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.726+0000 7f8e95bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f8e901402d0 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:15.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.726+0000 7f8e95bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f8e90149ae0 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:15.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.726+0000 7f8e95bcf640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e900718c0 con 0x7f8e90149ae0 2024-03-25T19:27:15.727 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.726+0000 7f8e95bcf640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e900714c0 con 0x7f8e900705b0 2024-03-25T19:27:15.727 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.726+0000 7f8e95bcf640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e90072b50 con 0x7f8e901402d0 2024-03-25T19:27:15.727 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.726+0000 7f8e95bcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:15.727 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.726+0000 7f8e95bcf640 10 monclient(hunting): authenticate will time out at 5873.086172s 2024-03-25T19:27:15.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.731+0000 7f8e8ffff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f8e90149ae0 0x7f8e9014bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:51564/0 (socket says 172.21.15.123:51564) 2024-03-25T19:27:15.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.731+0000 7f8e8ffff640 1 -- 172.21.15.123:0/2471230284 learned_addr learned my addr 172.21.15.123:0/2471230284 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:15.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.743+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052357840 0 0) 0x7f8e900718c0 con 0x7f8e90149ae0 2024-03-25T19:27:15.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.743+0000 7f8e8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.743+0000 7f8e8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.743+0000 7f8e8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.743+0000 7f8e8e7fc640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:27:15.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.743+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e78003520 con 0x7f8e90149ae0 2024-03-25T19:27:15.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.750+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204642948 0 0) 0x7f8e90072b50 con 0x7f8e901402d0 2024-03-25T19:27:15.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.750+0000 7f8e8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.750+0000 7f8e8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.750+0000 7f8e8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.750+0000 7f8e8e7fc640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:27:15.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.750+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e900718c0 con 0x7f8e901402d0 2024-03-25T19:27:15.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.758+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492832413 0 0) 0x7f8e900714c0 con 0x7f8e900705b0 2024-03-25T19:27:15.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.758+0000 7f8e8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.758+0000 7f8e8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.758+0000 7f8e8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.759 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.758+0000 7f8e8e7fc640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:27:15.759 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.758+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e90072b50 con 0x7f8e900705b0 2024-03-25T19:27:15.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005789698 0 0) 0x7f8e900718c0 con 0x7f8e901402d0 2024-03-25T19:27:15.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 >> v1:172.21.15.151:6789/0 conn(0x7f8e900705b0 legacy=0x7f8e9013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 >> v1:172.21.15.123:6789/0 conn(0x7f8e90149ae0 legacy=0x7f8e9014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:15.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:27:15.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9013dd10 con 0x7f8e901402d0 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8e84001670 con 0x7f8e901402d0 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e8e7fc640 10 monclient: dump: 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:15.761 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:15.762 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:15.762 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:15.762 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:15.762 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:15.762 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:15.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e95bcf640 5 monclient: authenticate success, global_id 4445 2024-03-25T19:27:15.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.760+0000 7f8e95bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:15.787 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.787+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e840017d0 con 0x7f8e901402d0 2024-03-25T19:27:15.787 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.787+0000 7f8e8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:15.787 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.787+0000 7f8e8e7fc640 1 -- 172.21.15.123:0/2471230284 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8e84001be0 con 0x7f8e901402d0 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.787+0000 7f8e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.787+0000 7f8e8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.787+0000 7f8e8e7fc640 10 monclient: dump: 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:15.788 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:15.789 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:15.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.787+0000 7f8e95bcf640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:15.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.787+0000 7f8e95bcf640 4 set_mon_vals no callback set 2024-03-25T19:27:15.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.788+0000 7f8e95bcf640 10 monclient: shutdown 2024-03-25T19:27:15.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.788+0000 7f8e95bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:15.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.788+0000 7f8e95bcf640 1 -- 172.21.15.123:0/2471230284 >> v1:172.21.15.123:6790/0 conn(0x7f8e901402d0 legacy=0x7f8e90140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.788+0000 7f8e95bcf640 1 -- 172.21.15.123:0/2471230284 shutdown_connections 2024-03-25T19:27:15.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.788+0000 7f8e95bcf640 1 -- 172.21.15.123:0/2471230284 >> 172.21.15.123:0/2471230284 conn(0x7f8e900b5d10 msgr2=0x7f8e900b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:15.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.788+0000 7f8e95bcf640 1 -- 172.21.15.123:0/2471230284 shutdown_connections 2024-03-25T19:27:15.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.788+0000 7f8e95bcf640 1 -- 172.21.15.123:0/2471230284 wait complete. 2024-03-25T19:27:15.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.789+0000 7f8e95bcf640 10 monclient: build_initial_monmap 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.789+0000 7f8e95bcf640 10 monclient: monmap: 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:15.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.789+0000 7f8e95bcf640 1 librados: starting msgr at 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.789+0000 7f8e95bcf640 1 librados: starting objecter 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.790+0000 7f8e95bcf640 1 Processor -- start 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.790+0000 7f8e95bcf640 1 -- start start 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.790+0000 7f8e95bcf640 1 librados: setting wanted keys 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.790+0000 7f8e95bcf640 1 librados: calling monclient init 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.790+0000 7f8e95bcf640 10 monclient: init 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.790+0000 7f8e95bcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.790+0000 7f8e95bcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.790+0000 7f8e95bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f8e900705b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.790+0000 7f8e95bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f8e901402d0 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.790+0000 7f8e95bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f8e90149ae0 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:15.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.791+0000 7f8e95bcf640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9013dfd0 con 0x7f8e90149ae0 2024-03-25T19:27:15.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.791+0000 7f8e95bcf640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e901430c0 con 0x7f8e900705b0 2024-03-25T19:27:15.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.791+0000 7f8e95bcf640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e901440c0 con 0x7f8e901402d0 2024-03-25T19:27:15.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.791+0000 7f8e95bcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:15.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.791+0000 7f8e95bcf640 10 monclient(hunting): authenticate will time out at 5873.150950s 2024-03-25T19:27:15.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.799+0000 7f8e8f7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f8e900705b0 0x7f8e90142440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:47164/0 (socket says 172.21.15.123:47164) 2024-03-25T19:27:15.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.799+0000 7f8e8f7fe640 1 -- 172.21.15.123:0/1392181281 learned_addr learned my addr 172.21.15.123:0/1392181281 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:15.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.802+0000 7f8e77fff640 10 client.?.objecter ms_handle_connect 0x7f8e900705b0 2024-03-25T19:27:15.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.802+0000 7f8e77fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:15.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.806+0000 7f8e77fff640 10 client.?.objecter ms_handle_connect 0x7f8e901402d0 2024-03-25T19:27:15.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.806+0000 7f8e77fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:15.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.808+0000 7f8e77fff640 10 client.?.objecter ms_handle_connect 0x7f8e90149ae0 2024-03-25T19:27:15.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.808+0000 7f8e77fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:15.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.810+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084876092 0 0) 0x7f8e90072b50 con 0x7f8e900705b0 2024-03-25T19:27:15.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.810+0000 7f8e77fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.810+0000 7f8e77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.810+0000 7f8e77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.810+0000 7f8e77fff640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:27:15.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.810+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e7c004eb0 con 0x7f8e900705b0 2024-03-25T19:27:15.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.816+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574364680 0 0) 0x7f8e9013dfd0 con 0x7f8e90149ae0 2024-03-25T19:27:15.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.816+0000 7f8e77fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.816+0000 7f8e77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.816+0000 7f8e77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.816+0000 7f8e77fff640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:27:15.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.816+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e90072b50 con 0x7f8e90149ae0 2024-03-25T19:27:15.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.818+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511379689 0 0) 0x7f8e901440c0 con 0x7f8e901402d0 2024-03-25T19:27:15.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.818+0000 7f8e77fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.818+0000 7f8e77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.818+0000 7f8e77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.818+0000 7f8e77fff640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:27:15.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.818+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9013dfd0 con 0x7f8e901402d0 2024-03-25T19:27:15.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.827+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1944926054 0 0) 0x7f8e7c004eb0 con 0x7f8e900705b0 2024-03-25T19:27:15.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.827+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e901440c0 con 0x7f8e900705b0 2024-03-25T19:27:15.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.827+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8e80001870 con 0x7f8e900705b0 2024-03-25T19:27:15.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.827+0000 7f8e77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:15.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.827+0000 7f8e77fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.827+0000 7f8e77fff640 10 monclient(hunting): dump: 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:15.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.827+0000 7f8e77fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:15.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2607121332 0 0) 0x7f8e901440c0 con 0x7f8e900705b0 2024-03-25T19:27:15.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 >> v1:172.21.15.123:6790/0 conn(0x7f8e901402d0 legacy=0x7f8e901d8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 >> v1:172.21.15.123:6789/0 conn(0x7f8e90149ae0 legacy=0x7f8e90148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:15.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 1 monclient: found mon.b 2024-03-25T19:27:15.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:15.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e90141eb0 con 0x7f8e900705b0 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e95bcf640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e95bcf640 10 monclient: _renew_subs 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e95bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e95bcf640 1 -- 172.21.15.123:0/1392181281 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e90142b50 con 0x7f8e900705b0 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e95bcf640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e95bcf640 10 monclient: _renew_subs 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e95bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:15.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e95bcf640 1 -- 172.21.15.123:0/1392181281 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e901d9830 con 0x7f8e900705b0 2024-03-25T19:27:15.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.839+0000 7f8e95bcf640 1 librados: init done 2024-03-25T19:27:15.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.840+0000 7f8e95bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:15.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.840+0000 7f8e95bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:15.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.840+0000 7f8e95bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:15.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:15.840+0000 7f8e95bcf640 1 -- 172.21.15.123:0/1392181281 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e900627b0 con 0x7f8e900705b0 2024-03-25T19:27:16.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.105+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e800035f0 con 0x7f8e900705b0 2024-03-25T19:27:16.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.105+0000 7f8e77fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.105+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8e80001d80 con 0x7f8e900705b0 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.105+0000 7f8e77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.105+0000 7f8e77fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.105+0000 7f8e77fff640 10 monclient: dump: 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:16.106 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:16.107 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:16.107 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:16.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.105+0000 7f8e8dffb640 4 set_mon_vals no callback set 2024-03-25T19:27:16.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.106+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f8e80003ac0 con 0x7f8e900705b0 2024-03-25T19:27:16.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.107+0000 7f8e8effd640 10 monclient: get_auth_request con 0x7f8e7c06ddf0 auth_method 0 2024-03-25T19:27:16.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.107+0000 7f8e77fff640 10 client.4426.objecter ms_handle_connect 0x7f8e7c06ddf0 2024-03-25T19:27:16.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.111+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f8e800546b0 con 0x7f8e900705b0 2024-03-25T19:27:16.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.111+0000 7f8e77fff640 10 client.4426.objecter ms_dispatch 0x7f8e90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:16.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.111+0000 7f8e77fff640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:16.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.111+0000 7f8e77fff640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:16.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.111+0000 7f8e77fff640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T19:27:16.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.111+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e800501c0 con 0x7f8e900705b0 2024-03-25T19:27:16.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.111+0000 7f8e77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:16.112 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.111+0000 7f8e77fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:16.367 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.367+0000 7f8e95bcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:16.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.367+0000 7f8e95bcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:16.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.367+0000 7f8e95bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:16.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.367+0000 7f8e95bcf640 1 -- 172.21.15.123:0/1392181281 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e90142ea0 con 0x7f8e900705b0 2024-03-25T19:27:16.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.397+0000 7f8e77fff640 1 -- 172.21.15.123:0/1392181281 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f8e800520f0 con 0x7f8e900705b0 2024-03-25T19:27:16.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.397+0000 7f8e77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:16.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.397+0000 7f8e77fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:16.398 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:27:16.398 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:16.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.399+0000 7f8e95bcf640 10 librados: watch_flush enter 2024-03-25T19:27:16.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.399+0000 7f8e95bcf640 10 librados: watch_flush exit 2024-03-25T19:27:16.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.399+0000 7f8e95bcf640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T19:27:16.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.399+0000 7f8e95bcf640 10 client.4426.objecter successfully canceled tick 2024-03-25T19:27:16.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.399+0000 7f8e95bcf640 1 -- 172.21.15.123:0/1392181281 >> v1:172.21.15.151:6800/727684239 conn(0x7f8e7c06ddf0 legacy=0x7f8e7c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:16.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.400+0000 7f8e95bcf640 10 monclient: shutdown 2024-03-25T19:27:16.400 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.400+0000 7f8e95bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:16.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.400+0000 7f8e95bcf640 1 -- 172.21.15.123:0/1392181281 >> v1:172.21.15.151:6789/0 conn(0x7f8e900705b0 legacy=0x7f8e90142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:16.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.400+0000 7f8e95bcf640 1 -- 172.21.15.123:0/1392181281 shutdown_connections 2024-03-25T19:27:16.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.400+0000 7f8e95bcf640 1 -- 172.21.15.123:0/1392181281 >> 172.21.15.123:0/1392181281 conn(0x7f8e900b5d10 msgr2=0x7f8e90140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:16.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.400+0000 7f8e95bcf640 1 -- 172.21.15.123:0/1392181281 shutdown_connections 2024-03-25T19:27:16.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.400+0000 7f8e95bcf640 1 -- 172.21.15.123:0/1392181281 wait complete. 2024-03-25T19:27:16.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:16.400+0000 7f8e95bcf640 1 librados: shutdown 2024-03-25T19:27:16.412 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:23.219 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.332+0000 7fefa4484640 10 monclient: get_monmap_and_config 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.333+0000 7fefa4484640 10 monclient: build_initial_monmap 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.333+0000 7fefa4484640 10 monclient: monmap: 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:27:23.334932+0000 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:27:23.334932+0000 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:27:23.334 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:27:23.335 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:23.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.333+0000 7fefa4484640 1 Processor -- start 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.333+0000 7fefa4484640 1 -- start start 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.333+0000 7fefa4484640 10 monclient: init 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.334+0000 7fefa4484640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.334+0000 7fefa4484640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.334+0000 7fefa4484640 10 monclient(hunting): picked mon.noname-c con 0x7fef9c0a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.334+0000 7fefa4484640 10 monclient(hunting): picked mon.noname-b con 0x7fef9c13f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.334+0000 7fefa4484640 10 monclient(hunting): picked mon.noname-a con 0x7fef9c1430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.334+0000 7fefa4484640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c142520 con 0x7fef9c1430c0 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.334+0000 7fefa4484640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c0b28a0 con 0x7fef9c0a98f0 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.335+0000 7fefa4484640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c0b2a30 con 0x7fef9c13f710 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.335+0000 7fefa4484640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:23.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.335+0000 7fefa4484640 10 monclient(hunting): authenticate will time out at 5880.695889s 2024-03-25T19:27:23.337 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.336+0000 7fefa1a1f640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fef9c13f710 0x7fef9c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:57192/0 (socket says 172.21.15.123:57192) 2024-03-25T19:27:23.337 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.336+0000 7fefa1a1f640 1 -- 172.21.15.123:0/4258763782 learned_addr learned my addr 172.21.15.123:0/4258763782 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:23.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.349+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942808128 0 0) 0x7fef9c0b2a30 con 0x7fef9c13f710 2024-03-25T19:27:23.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.349+0000 7fefa121e640 10 monclient(hunting): _negotiate 2024-03-25T19:27:23.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.349+0000 7fefa121e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:23.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.349+0000 7fefa121e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:23.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.349+0000 7fefa121e640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:27:23.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.349+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef88003520 con 0x7fef9c13f710 2024-03-25T19:27:23.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.352+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689868363 0 0) 0x7fef9c0b28a0 con 0x7fef9c0a98f0 2024-03-25T19:27:23.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.352+0000 7fefa121e640 10 monclient(hunting): _negotiate 2024-03-25T19:27:23.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.352+0000 7fefa121e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:23.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.352+0000 7fefa121e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:23.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.352+0000 7fefa121e640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:27:23.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.352+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c0b2a30 con 0x7fef9c0a98f0 2024-03-25T19:27:23.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3980775507 0 0) 0x7fef88003520 con 0x7fef9c13f710 2024-03-25T19:27:23.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 >> v1:172.21.15.151:6789/0 conn(0x7fef9c0a98f0 legacy=0x7fef9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:23.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 >> v1:172.21.15.123:6789/0 conn(0x7fef9c1430c0 legacy=0x7fef9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:23.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 10 monclient: _finish_hunting 0 2024-03-25T19:27:23.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 1 monclient: found mon.noname-b 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c0b44c0 con 0x7fef9c13f710 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 10 monclient: _finish_auth 0 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 10 monclient: _check_auth_tickets 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fef8c001690 con 0x7fef9c13f710 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa121e640 10 monclient: dump: 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:23.364 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:23.365 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:23.365 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:23.365 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:23.365 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:23.365 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:23.365 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:23.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa4484640 5 monclient: authenticate success, global_id 4451 2024-03-25T19:27:23.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.362+0000 7fefa4484640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:23.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.372+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef8c001870 con 0x7fef9c13f710 2024-03-25T19:27:23.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.372+0000 7fefa121e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:23.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.372+0000 7fefa121e640 1 -- 172.21.15.123:0/4258763782 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fef8c001c80 con 0x7fef9c13f710 2024-03-25T19:27:23.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.372+0000 7fefa4484640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:23.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.372+0000 7fefa4484640 4 set_mon_vals no callback set 2024-03-25T19:27:23.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.372+0000 7fefa121e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:23.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.372+0000 7fefa121e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:23.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.372+0000 7fefa121e640 10 monclient: dump: 2024-03-25T19:27:23.373 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:23.373 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:23.374 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:23.374 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:23.374 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:23.374 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:23.374 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:23.374 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:23.374 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:23.374 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:23.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.373+0000 7fefa4484640 10 monclient: shutdown 2024-03-25T19:27:23.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.373+0000 7fefa4484640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:23.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.373+0000 7fefa4484640 1 -- 172.21.15.123:0/4258763782 >> v1:172.21.15.123:6790/0 conn(0x7fef9c13f710 legacy=0x7fef9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:23.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.373+0000 7fefa4484640 1 -- 172.21.15.123:0/4258763782 shutdown_connections 2024-03-25T19:27:23.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.373+0000 7fefa4484640 1 -- 172.21.15.123:0/4258763782 >> 172.21.15.123:0/4258763782 conn(0x7fef9c0b7ea0 msgr2=0x7fef9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:23.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.373+0000 7fefa4484640 1 -- 172.21.15.123:0/4258763782 shutdown_connections 2024-03-25T19:27:23.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.373+0000 7fefa4484640 1 -- 172.21.15.123:0/4258763782 wait complete. 2024-03-25T19:27:23.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.374+0000 7fefa4484640 10 monclient: build_initial_monmap 2024-03-25T19:27:23.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.374+0000 7fefa4484640 10 monclient: monmap: 2024-03-25T19:27:23.376 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.374+0000 7fefa4484640 1 librados: starting msgr at 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.374+0000 7fefa4484640 1 librados: starting objecter 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.374+0000 7fefa4484640 1 Processor -- start 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.374+0000 7fefa4484640 1 -- start start 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.374+0000 7fefa4484640 1 librados: setting wanted keys 2024-03-25T19:27:23.377 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.374+0000 7fefa4484640 1 librados: calling monclient init 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.374+0000 7fefa4484640 10 monclient: init 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.375+0000 7fefa4484640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.375+0000 7fefa4484640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.375+0000 7fefa4484640 10 monclient(hunting): picked mon.noname-a con 0x7fef9c0a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.375+0000 7fefa4484640 10 monclient(hunting): picked mon.noname-b con 0x7fef9c13f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.375+0000 7fefa4484640 10 monclient(hunting): picked mon.noname-c con 0x7fef9c1430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.375+0000 7fefa4484640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c13be70 con 0x7fef9c0a98f0 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.375+0000 7fefa4484640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c131660 con 0x7fef9c13f710 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.375+0000 7fefa4484640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c1317f0 con 0x7fef9c1430c0 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.375+0000 7fefa4484640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:23.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.375+0000 7fefa4484640 10 monclient(hunting): authenticate will time out at 5880.736290s 2024-03-25T19:27:23.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.380+0000 7fefa2220640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fef9c0a98f0 0x7fef9c130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:56960/0 (socket says 172.21.15.123:56960) 2024-03-25T19:27:23.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.380+0000 7fefa2220640 1 -- 172.21.15.123:0/776124206 learned_addr learned my addr 172.21.15.123:0/776124206 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:23.387 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.386+0000 7fef92ffd640 10 client.?.objecter ms_handle_connect 0x7fef9c1430c0 2024-03-25T19:27:23.387 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.386+0000 7fef92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:23.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.387+0000 7fef92ffd640 10 client.?.objecter ms_handle_connect 0x7fef9c0a98f0 2024-03-25T19:27:23.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.387+0000 7fef92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:23.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.392+0000 7fef92ffd640 10 client.?.objecter ms_handle_connect 0x7fef9c13f710 2024-03-25T19:27:23.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.392+0000 7fef92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:23.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.395+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750611562 0 0) 0x7fef9c1317f0 con 0x7fef9c1430c0 2024-03-25T19:27:23.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.395+0000 7fef92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:23.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.395+0000 7fef92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:23.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.395+0000 7fef92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:23.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.395+0000 7fef92ffd640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:27:23.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.395+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef80004eb0 con 0x7fef9c1430c0 2024-03-25T19:27:23.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.396+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325966648 0 0) 0x7fef9c13be70 con 0x7fef9c0a98f0 2024-03-25T19:27:23.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.396+0000 7fef92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:23.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.396+0000 7fef92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:23.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.396+0000 7fef92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:23.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.396+0000 7fef92ffd640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:27:23.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.396+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c1317f0 con 0x7fef9c0a98f0 2024-03-25T19:27:23.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.405+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1227773828 0 0) 0x7fef9c1317f0 con 0x7fef9c0a98f0 2024-03-25T19:27:23.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.405+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef9c13be70 con 0x7fef9c0a98f0 2024-03-25T19:27:23.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.405+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fef980019f0 con 0x7fef9c0a98f0 2024-03-25T19:27:23.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.405+0000 7fef92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:23.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.405+0000 7fef92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:23.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.405+0000 7fef92ffd640 10 monclient(hunting): dump: 2024-03-25T19:27:23.406 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:23.406 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:23.407 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:23.407 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:23.407 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:23.407 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:23.407 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:23.407 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:23.407 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:23.407 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:23.407 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.405+0000 7fef92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:23.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.409+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 605884809 0 0) 0x7fef80004eb0 con 0x7fef9c1430c0 2024-03-25T19:27:23.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.409+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef9c1317f0 con 0x7fef9c1430c0 2024-03-25T19:27:23.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.409+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fef94001b90 con 0x7fef9c1430c0 2024-03-25T19:27:23.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.409+0000 7fef92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:23.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.409+0000 7fef92ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:23.410 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.409+0000 7fef92ffd640 10 monclient(hunting): dump: 2024-03-25T19:27:23.410 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:23.410 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:23.410 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:23.410 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:23.411 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:23.411 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:23.411 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:23.411 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:23.411 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:23.411 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3348487170 0 0) 0x7fef9c13be70 con 0x7fef9c0a98f0 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 >> v1:172.21.15.123:6790/0 conn(0x7fef9c1430c0 legacy=0x7fef9c1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 >> v1:172.21.15.151:6789/0 conn(0x7fef9c13f710 legacy=0x7fef9c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 1 monclient: found mon.a 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c131980 con 0x7fef9c0a98f0 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 10 monclient: _finish_auth 0 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:23.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fef92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:23.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fefa4484640 5 monclient: authenticate success, global_id 4425 2024-03-25T19:27:23.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fefa4484640 10 monclient: _renew_subs 2024-03-25T19:27:23.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fefa4484640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:23.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fefa4484640 1 -- 172.21.15.123:0/776124206 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c135c60 con 0x7fef9c0a98f0 2024-03-25T19:27:23.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fefa4484640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:23.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fefa4484640 10 monclient: _renew_subs 2024-03-25T19:27:23.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fefa4484640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:23.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fefa4484640 1 -- 172.21.15.123:0/776124206 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c0b7770 con 0x7fef9c0a98f0 2024-03-25T19:27:23.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.415+0000 7fefa4484640 1 librados: init done 2024-03-25T19:27:23.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.416+0000 7fefa4484640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:23.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.416+0000 7fefa4484640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:23.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.416+0000 7fefa4484640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:23.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.416+0000 7fefa4484640 1 -- 172.21.15.123:0/776124206 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c1371d0 con 0x7fef9c0a98f0 2024-03-25T19:27:23.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.741+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef980035f0 con 0x7fef9c0a98f0 2024-03-25T19:27:23.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.741+0000 7fef92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.741+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fef980039c0 con 0x7fef9c0a98f0 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.741+0000 7fef92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.741+0000 7fef92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.741+0000 7fef92ffd640 10 monclient: dump: 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:23.743 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:23.744 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:23.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.741+0000 7fefa0a1d640 4 set_mon_vals no callback set 2024-03-25T19:27:23.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.742+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fef98003ba0 con 0x7fef9c0a98f0 2024-03-25T19:27:23.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.742+0000 7fefa1a1f640 10 monclient: get_auth_request con 0x7fef8006dfb0 auth_method 0 2024-03-25T19:27:23.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.743+0000 7fef92ffd640 10 client.4425.objecter ms_handle_connect 0x7fef8006dfb0 2024-03-25T19:27:23.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.748+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fef98054790 con 0x7fef9c0a98f0 2024-03-25T19:27:23.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.748+0000 7fef92ffd640 10 client.4425.objecter ms_dispatch 0x7fef9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:23.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.748+0000 7fef92ffd640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:23.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.748+0000 7fef92ffd640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:23.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.748+0000 7fef92ffd640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-25T19:27:23.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.748+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef980502a0 con 0x7fef9c0a98f0 2024-03-25T19:27:23.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.748+0000 7fef92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:23.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.748+0000 7fef92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:23.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.966+0000 7fefa4484640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:23.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.966+0000 7fefa4484640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:23.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.966+0000 7fefa4484640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:23.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:23.966+0000 7fefa4484640 1 -- 172.21.15.123:0/776124206 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef9c074cc0 con 0x7fef9c0a98f0 2024-03-25T19:27:24.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.009+0000 7fef92ffd640 1 -- 172.21.15.123:0/776124206 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fef980521d0 con 0x7fef9c0a98f0 2024-03-25T19:27:24.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.009+0000 7fef92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:24.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.009+0000 7fef92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:24.010 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:27:24.011 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:24.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.011+0000 7fefa4484640 10 librados: watch_flush enter 2024-03-25T19:27:24.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 10 librados: watch_flush exit 2024-03-25T19:27:24.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-25T19:27:24.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 10 client.4425.objecter successfully canceled tick 2024-03-25T19:27:24.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 1 -- 172.21.15.123:0/776124206 >> v1:172.21.15.151:6800/727684239 conn(0x7fef8006dfb0 legacy=0x7fef80070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:24.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 10 monclient: shutdown 2024-03-25T19:27:24.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:24.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 1 -- 172.21.15.123:0/776124206 >> v1:172.21.15.123:6789/0 conn(0x7fef9c0a98f0 legacy=0x7fef9c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:24.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 1 -- 172.21.15.123:0/776124206 shutdown_connections 2024-03-25T19:27:24.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 1 -- 172.21.15.123:0/776124206 >> 172.21.15.123:0/776124206 conn(0x7fef9c0b7ea0 msgr2=0x7fef9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:24.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 1 -- 172.21.15.123:0/776124206 shutdown_connections 2024-03-25T19:27:24.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 1 -- 172.21.15.123:0/776124206 wait complete. 2024-03-25T19:27:24.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:24.012+0000 7fefa4484640 1 librados: shutdown 2024-03-25T19:27:24.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:30.606 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:05:20.178 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:30.614 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:05:20.187 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:30.649 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:05:20.220 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:30.657 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:05:20.229 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:30.662 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:05:20.196 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:30.666 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:05:20.199 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:30.666 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:05:20.238 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:30.697 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:05:20.269 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:30.699 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:05:20.233 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:30.832 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:30.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.944+0000 7f75738af640 10 monclient: get_monmap_and_config 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.944+0000 7f75738af640 10 monclient: build_initial_monmap 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.944+0000 7f75738af640 10 monclient: monmap: 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:27:30.945868+0000 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:27:30.945868+0000 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:27:30.945 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.945+0000 7f75738af640 1 Processor -- start 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.945+0000 7f75738af640 1 -- start start 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.945+0000 7f75738af640 10 monclient: init 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.946+0000 7f75738af640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.946+0000 7f75738af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.946+0000 7f75738af640 10 monclient(hunting): picked mon.noname-a con 0x7f756c0b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.946+0000 7f75738af640 10 monclient(hunting): picked mon.noname-c con 0x7f756c0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.946+0000 7f75738af640 10 monclient(hunting): picked mon.noname-b con 0x7f756c14be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.946+0000 7f75738af640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c0af940 con 0x7f756c0b4d60 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.946+0000 7f75738af640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c0b0a70 con 0x7f756c0b1730 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.946+0000 7f75738af640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c0b0d50 con 0x7f756c14be90 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.946+0000 7f75738af640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:30.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.946+0000 7f75738af640 10 monclient(hunting): authenticate will time out at 5888.306196s 2024-03-25T19:27:30.972 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.972+0000 7f7570e4a640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f756c0b1730 0x7f756c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:57774/0 (socket says 172.21.15.123:57774) 2024-03-25T19:27:30.972 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.972+0000 7f7570e4a640 1 -- 172.21.15.123:0/2343212025 learned_addr learned my addr 172.21.15.123:0/2343212025 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:30.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.991+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225189792 0 0) 0x7f756c0b0d50 con 0x7f756c14be90 2024-03-25T19:27:30.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.991+0000 7f7563fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:30.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.991+0000 7f7563fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:30.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.991+0000 7f7563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:30.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.991+0000 7f7563fff640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:27:30.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.991+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7558003520 con 0x7f756c14be90 2024-03-25T19:27:30.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.992+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141154592 0 0) 0x7f756c0b0a70 con 0x7f756c0b1730 2024-03-25T19:27:30.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.992+0000 7f7563fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:30.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.992+0000 7f7563fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:30.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.992+0000 7f7563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:30.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.992+0000 7f7563fff640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:27:30.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:30.992+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c0b0d50 con 0x7f756c0b1730 2024-03-25T19:27:31.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1403874726 0 0) 0x7f756c0b0d50 con 0x7f756c0b1730 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 >> v1:172.21.15.123:6790/0 conn(0x7f756c14be90 legacy=0x7f756c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 >> v1:172.21.15.123:6789/0 conn(0x7f756c0b4d60 legacy=0x7f756c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 1 monclient: found mon.noname-c 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c0b1030 con 0x7f756c0b1730 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f755c001670 con 0x7f756c0b1730 2024-03-25T19:27:31.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f7563fff640 10 monclient: dump: 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:31.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f75738af640 5 monclient: authenticate success, global_id 4435 2024-03-25T19:27:31.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.008+0000 7f75738af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:31.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f755c0017d0 con 0x7f756c0b1730 2024-03-25T19:27:31.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f7563fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f7563fff640 1 -- 172.21.15.123:0/2343212025 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f755c001ba0 con 0x7f756c0b1730 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f75738af640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f75738af640 4 set_mon_vals no callback set 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f7563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f7563fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f7563fff640 10 monclient: dump: 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:31.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f75738af640 10 monclient: shutdown 2024-03-25T19:27:31.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f75738af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:31.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f75738af640 1 -- 172.21.15.123:0/2343212025 >> v1:172.21.15.151:6789/0 conn(0x7f756c0b1730 legacy=0x7f756c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:31.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f75738af640 1 -- 172.21.15.123:0/2343212025 shutdown_connections 2024-03-25T19:27:31.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.028+0000 7f75738af640 1 -- 172.21.15.123:0/2343212025 >> 172.21.15.123:0/2343212025 conn(0x7f756c0617a0 msgr2=0x7f756c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:31.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.029+0000 7f75738af640 1 -- 172.21.15.123:0/2343212025 shutdown_connections 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.029+0000 7f75738af640 1 -- 172.21.15.123:0/2343212025 wait complete. 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.029+0000 7f75738af640 10 monclient: build_initial_monmap 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.029+0000 7f75738af640 10 monclient: monmap: 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.030+0000 7f75738af640 1 librados: starting msgr at 2024-03-25T19:27:31.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.030+0000 7f75738af640 1 librados: starting objecter 2024-03-25T19:27:31.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.030+0000 7f75738af640 1 Processor -- start 2024-03-25T19:27:31.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.030+0000 7f75738af640 1 -- start start 2024-03-25T19:27:31.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.030+0000 7f75738af640 1 librados: setting wanted keys 2024-03-25T19:27:31.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.030+0000 7f75738af640 1 librados: calling monclient init 2024-03-25T19:27:31.032 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.030+0000 7f75738af640 10 monclient: init 2024-03-25T19:27:31.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.031+0000 7f75738af640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:31.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.031+0000 7f75738af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:31.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.031+0000 7f75738af640 10 monclient(hunting): picked mon.noname-a con 0x7f756c0b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:31.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.031+0000 7f75738af640 10 monclient(hunting): picked mon.noname-c con 0x7f756c0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:31.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.031+0000 7f75738af640 10 monclient(hunting): picked mon.noname-b con 0x7f756c14be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:31.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.031+0000 7f75738af640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c1dacc0 con 0x7f756c0b1730 2024-03-25T19:27:31.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.031+0000 7f75738af640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c142850 con 0x7f756c14be90 2024-03-25T19:27:31.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.031+0000 7f75738af640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c1429e0 con 0x7f756c0b4d60 2024-03-25T19:27:31.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.031+0000 7f75738af640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:31.034 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.031+0000 7f75738af640 10 monclient(hunting): authenticate will time out at 5888.391323s 2024-03-25T19:27:31.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.038+0000 7f7571e4c640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f756c14be90 0x7f756c13b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:57836/0 (socket says 172.21.15.123:57836) 2024-03-25T19:27:31.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.038+0000 7f7571e4c640 1 -- 172.21.15.123:0/72201365 learned_addr learned my addr 172.21.15.123:0/72201365 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:31.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.038+0000 7f7561ffb640 10 client.?.objecter ms_handle_connect 0x7f756c14be90 2024-03-25T19:27:31.038 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.038+0000 7f7561ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:31.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.043+0000 7f7561ffb640 10 client.?.objecter ms_handle_connect 0x7f756c0b4d60 2024-03-25T19:27:31.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.043+0000 7f7561ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:31.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.046+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302920959 0 0) 0x7f756c142850 con 0x7f756c14be90 2024-03-25T19:27:31.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.046+0000 7f7561ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:31.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.046+0000 7f7561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:31.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.046+0000 7f7561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:31.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.046+0000 7f7561ffb640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:27:31.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.046+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7550004eb0 con 0x7f756c14be90 2024-03-25T19:27:31.056 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.056+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2330906358 0 0) 0x7f7550004eb0 con 0x7f756c14be90 2024-03-25T19:27:31.056 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.056+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f756c142850 con 0x7f756c14be90 2024-03-25T19:27:31.056 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.056+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7564001c30 con 0x7f756c14be90 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.056+0000 7f7561ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.056+0000 7f7561ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.056+0000 7f7561ffb640 10 monclient(hunting): dump: 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:31.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.056+0000 7f7561ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:31.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.061+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965240600 0 0) 0x7f756c0b1030 con 0x7f756c0b4d60 2024-03-25T19:27:31.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.061+0000 7f7561ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:31.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.061+0000 7f7561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:31.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.061+0000 7f7561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:31.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.061+0000 7f7561ffb640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:27:31.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.061+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7550004eb0 con 0x7f756c0b4d60 2024-03-25T19:27:31.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.063+0000 7f7561ffb640 10 client.?.objecter ms_handle_connect 0x7f756c0b1730 2024-03-25T19:27:31.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.063+0000 7f7561ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:31.068 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1923324880 0 0) 0x7f756c142850 con 0x7f756c14be90 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 >> v1:172.21.15.123:6790/0 conn(0x7f756c0b4d60 legacy=0x7f756c1db1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 >> v1:172.21.15.123:6789/0 conn(0x7f756c0b1730 legacy=0x7f756c139160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 1 monclient: found mon.b 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c1db8f0 con 0x7f756c14be90 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 10 monclient: _finish_auth 0 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f7561ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f75738af640 5 monclient: authenticate success, global_id 4438 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f75738af640 10 monclient: _renew_subs 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f75738af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f75738af640 1 -- 172.21.15.123:0/72201365 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756c1430d0 con 0x7f756c14be90 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f75738af640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:31.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f75738af640 10 monclient: _renew_subs 2024-03-25T19:27:31.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.068+0000 7f75738af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:31.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.069+0000 7f75738af640 1 -- 172.21.15.123:0/72201365 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f756c13aea0 con 0x7f756c14be90 2024-03-25T19:27:31.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.069+0000 7f75738af640 1 librados: init done 2024-03-25T19:27:31.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.070+0000 7f75738af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:31.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.070+0000 7f75738af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:31.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.070+0000 7f75738af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:31.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.070+0000 7f75738af640 1 -- 172.21.15.123:0/72201365 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756c1dab20 con 0x7f756c14be90 2024-03-25T19:27:31.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.358+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7564003990 con 0x7f756c14be90 2024-03-25T19:27:31.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.358+0000 7f7561ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:31.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.358+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7564003c00 con 0x7f756c14be90 2024-03-25T19:27:31.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.358+0000 7f7561ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:31.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.358+0000 7f7561ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:31.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.358+0000 7f7561ffb640 10 monclient: dump: 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:31.359 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.358+0000 7f75637fe640 4 set_mon_vals no callback set 2024-03-25T19:27:31.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.359+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f7564003e20 con 0x7f756c14be90 2024-03-25T19:27:31.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.360+0000 7f757164b640 10 monclient: get_auth_request con 0x7f755006de20 auth_method 0 2024-03-25T19:27:31.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.360+0000 7f7561ffb640 10 client.4438.objecter ms_handle_connect 0x7f755006de20 2024-03-25T19:27:31.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.365+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f7564054830 con 0x7f756c14be90 2024-03-25T19:27:31.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.365+0000 7f7561ffb640 10 client.4438.objecter ms_dispatch 0x7f756c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:31.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.365+0000 7f7561ffb640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:31.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.365+0000 7f7561ffb640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:31.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.365+0000 7f7561ffb640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-25T19:27:31.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.365+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7564050340 con 0x7f756c14be90 2024-03-25T19:27:31.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.365+0000 7f7561ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:31.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.365+0000 7f7561ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:31.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.617+0000 7f75738af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:31.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.617+0000 7f75738af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:31.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.617+0000 7f75738af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:27:31.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.617+0000 7f75738af640 1 -- 172.21.15.123:0/72201365 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f756c1dba80 con 0x7f756c14be90 2024-03-25T19:27:31.701 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.701+0000 7f7561ffb640 1 -- 172.21.15.123:0/72201365 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f7564052270 con 0x7f756c14be90 2024-03-25T19:27:31.701 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.701+0000 7f7561ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:31.701 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.701+0000 7f7561ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:31.701 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:27:31.701 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:31.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.703+0000 7f75738af640 10 librados: watch_flush enter 2024-03-25T19:27:31.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 10 librados: watch_flush exit 2024-03-25T19:27:31.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-25T19:27:31.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 10 client.4438.objecter successfully canceled tick 2024-03-25T19:27:31.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 1 -- 172.21.15.123:0/72201365 >> v1:172.21.15.151:6800/727684239 conn(0x7f755006de20 legacy=0x7f75500702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:31.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 10 monclient: shutdown 2024-03-25T19:27:31.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:31.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 1 -- 172.21.15.123:0/72201365 >> v1:172.21.15.151:6789/0 conn(0x7f756c14be90 legacy=0x7f756c13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:31.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 1 -- 172.21.15.123:0/72201365 shutdown_connections 2024-03-25T19:27:31.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 1 -- 172.21.15.123:0/72201365 >> 172.21.15.123:0/72201365 conn(0x7f756c0617a0 msgr2=0x7f756c137300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:31.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 1 -- 172.21.15.123:0/72201365 shutdown_connections 2024-03-25T19:27:31.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 1 -- 172.21.15.123:0/72201365 wait complete. 2024-03-25T19:27:31.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:31.704+0000 7f75738af640 1 librados: shutdown 2024-03-25T19:27:31.716 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:38.522 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:38.636 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.636+0000 7f12adf54640 10 monclient: get_monmap_and_config 2024-03-25T19:27:38.636 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.636+0000 7f12adf54640 10 monclient: build_initial_monmap 2024-03-25T19:27:38.636 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.636+0000 7f12adf54640 10 monclient: monmap: 2024-03-25T19:27:38.636 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:38.636 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:38.636 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:27:38.637142+0000 2024-03-25T19:27:38.636 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:27:38.637142+0000 2024-03-25T19:27:38.636 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:38.636 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:38.636 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:38.637 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:27:38.637 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:27:38.637 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:38.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 1 Processor -- start 2024-03-25T19:27:38.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 1 -- start start 2024-03-25T19:27:38.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 10 monclient: init 2024-03-25T19:27:38.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:38.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:27:38.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 10 monclient(hunting): picked mon.noname-b con 0x7f12a80b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:38.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 10 monclient(hunting): picked mon.noname-c con 0x7f12a80b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:38.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 10 monclient(hunting): picked mon.noname-a con 0x7f12a814be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:38.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a80af940 con 0x7f12a814be90 2024-03-25T19:27:38.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a80b0a70 con 0x7f12a80b1730 2024-03-25T19:27:38.639 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a80b0d50 con 0x7f12a80b4d60 2024-03-25T19:27:38.639 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:38.639 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.637+0000 7f12adf54640 10 monclient(hunting): authenticate will time out at 5895.997729s 2024-03-25T19:27:38.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.640+0000 7f12a77fe640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f12a80b4d60 0x7f12a813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:49570/0 (socket says 172.21.15.123:49570) 2024-03-25T19:27:38.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.640+0000 7f12a77fe640 1 -- 172.21.15.123:0/3073008123 learned_addr learned my addr 172.21.15.123:0/3073008123 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:38.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.663+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999919388 0 0) 0x7f12a80af940 con 0x7f12a814be90 2024-03-25T19:27:38.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.663+0000 7f12a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.663+0000 7f12a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.663+0000 7f12a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.663+0000 7f12a67fc640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:27:38.663 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.663+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1290003520 con 0x7f12a814be90 2024-03-25T19:27:38.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.669+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693187648 0 0) 0x7f12a80b0d50 con 0x7f12a80b4d60 2024-03-25T19:27:38.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.669+0000 7f12a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.669+0000 7f12a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.669+0000 7f12a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.669+0000 7f12a67fc640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:27:38.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.669+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a80af940 con 0x7f12a80b4d60 2024-03-25T19:27:38.677 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.677+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647714267 0 0) 0x7f12a80b0a70 con 0x7f12a80b1730 2024-03-25T19:27:38.677 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.677+0000 7f12a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.677 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.677+0000 7f12a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.677 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.677+0000 7f12a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.677 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.677+0000 7f12a67fc640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:27:38.677 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.677+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a80b0d50 con 0x7f12a80b1730 2024-03-25T19:27:38.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 575538151 0 0) 0x7f12a80af940 con 0x7f12a80b4d60 2024-03-25T19:27:38.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 >> v1:172.21.15.151:6789/0 conn(0x7f12a80b1730 legacy=0x7f12a80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 >> v1:172.21.15.123:6789/0 conn(0x7f12a814be90 legacy=0x7f12a814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 1 monclient: found mon.noname-b 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a80b1030 con 0x7f12a80b4d60 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12adf54640 5 monclient: authenticate success, global_id 4463 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1298001670 con 0x7f12a80b4d60 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12adf54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:38.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12a67fc640 10 monclient: dump: 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:38.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.683+0000 7f12adf54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:38.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.703+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12980017d0 con 0x7f12a80b4d60 2024-03-25T19:27:38.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.703+0000 7f12a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:38.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.703+0000 7f12a67fc640 1 -- 172.21.15.123:0/3073008123 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1298001be0 con 0x7f12a80b4d60 2024-03-25T19:27:38.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.703+0000 7f12adf54640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:38.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.703+0000 7f12adf54640 4 set_mon_vals no callback set 2024-03-25T19:27:38.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.703+0000 7f12a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.703+0000 7f12a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.703+0000 7f12a67fc640 10 monclient: dump: 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:38.704 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:38.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.704+0000 7f12adf54640 10 monclient: shutdown 2024-03-25T19:27:38.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.704+0000 7f12adf54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:38.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.704+0000 7f12adf54640 1 -- 172.21.15.123:0/3073008123 >> v1:172.21.15.123:6790/0 conn(0x7f12a80b4d60 legacy=0x7f12a813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.704+0000 7f12adf54640 1 -- 172.21.15.123:0/3073008123 shutdown_connections 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.704+0000 7f12adf54640 1 -- 172.21.15.123:0/3073008123 >> 172.21.15.123:0/3073008123 conn(0x7f12a80617a0 msgr2=0x7f12a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.704+0000 7f12adf54640 1 -- 172.21.15.123:0/3073008123 shutdown_connections 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.704+0000 7f12adf54640 1 -- 172.21.15.123:0/3073008123 wait complete. 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.705+0000 7f12adf54640 10 monclient: build_initial_monmap 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.705+0000 7f12adf54640 10 monclient: monmap: 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:27:38.706 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:27:38.707 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:38.707 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.705+0000 7f12adf54640 1 librados: starting msgr at 2024-03-25T19:27:38.707 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.705+0000 7f12adf54640 1 librados: starting objecter 2024-03-25T19:27:38.707 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.706+0000 7f12adf54640 1 Processor -- start 2024-03-25T19:27:38.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.706+0000 7f12adf54640 1 -- start start 2024-03-25T19:27:38.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.706+0000 7f12adf54640 1 librados: setting wanted keys 2024-03-25T19:27:38.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.706+0000 7f12adf54640 1 librados: calling monclient init 2024-03-25T19:27:38.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.706+0000 7f12adf54640 10 monclient: init 2024-03-25T19:27:38.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.706+0000 7f12adf54640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:38.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.706+0000 7f12adf54640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:38.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.706+0000 7f12adf54640 10 monclient(hunting): picked mon.noname-a con 0x7f12a80b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:38.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.706+0000 7f12adf54640 10 monclient(hunting): picked mon.noname-b con 0x7f12a80b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:38.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.707+0000 7f12adf54640 10 monclient(hunting): picked mon.noname-c con 0x7f12a814be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:38.709 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.707+0000 7f12adf54640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a813e200 con 0x7f12a80b1730 2024-03-25T19:27:38.709 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.707+0000 7f12adf54640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a812f5f0 con 0x7f12a80b4d60 2024-03-25T19:27:38.709 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.707+0000 7f12adf54640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a812f780 con 0x7f12a814be90 2024-03-25T19:27:38.709 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.707+0000 7f12adf54640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:38.709 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.707+0000 7f12adf54640 10 monclient(hunting): authenticate will time out at 5896.067040s 2024-03-25T19:27:38.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.719+0000 7f12a6ffd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f12a80b4d60 0x7f12a812eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:52214/0 (socket says 172.21.15.123:52214) 2024-03-25T19:27:38.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.719+0000 7f12a6ffd640 1 -- 172.21.15.123:0/1369157841 learned_addr learned my addr 172.21.15.123:0/1369157841 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:38.720 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.721+0000 7f128ffff640 10 client.?.objecter ms_handle_connect 0x7f12a80b4d60 2024-03-25T19:27:38.721 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.721+0000 7f128ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:38.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.728+0000 7f128ffff640 10 client.?.objecter ms_handle_connect 0x7f12a814be90 2024-03-25T19:27:38.728 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.728+0000 7f128ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:38.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.729+0000 7f128ffff640 10 client.?.objecter ms_handle_connect 0x7f12a80b1730 2024-03-25T19:27:38.729 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.729+0000 7f128ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:38.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.735+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643161781 0 0) 0x7f12a812f5f0 con 0x7f12a80b4d60 2024-03-25T19:27:38.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.735+0000 7f128ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.735+0000 7f128ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.735+0000 7f128ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.735+0000 7f128ffff640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:27:38.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.735+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1294004eb0 con 0x7f12a80b4d60 2024-03-25T19:27:38.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.743+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989911629 0 0) 0x7f12a813e200 con 0x7f12a80b1730 2024-03-25T19:27:38.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.743+0000 7f128ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.743+0000 7f128ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.743+0000 7f128ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.743+0000 7f128ffff640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:27:38.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.743+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a812f5f0 con 0x7f12a80b1730 2024-03-25T19:27:38.753 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.753+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552548523 0 0) 0x7f12a812f780 con 0x7f12a814be90 2024-03-25T19:27:38.753 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.753+0000 7f128ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.753 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.753+0000 7f128ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.753+0000 7f128ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.753+0000 7f128ffff640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:27:38.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.753+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a813e200 con 0x7f12a814be90 2024-03-25T19:27:38.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.761+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4003497440 0 0) 0x7f12a812f5f0 con 0x7f12a80b1730 2024-03-25T19:27:38.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.761+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12a812f780 con 0x7f12a80b1730 2024-03-25T19:27:38.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.761+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1298001ae0 con 0x7f12a80b1730 2024-03-25T19:27:38.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.761+0000 7f128ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:38.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.761+0000 7f128ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:38.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.761+0000 7f128ffff640 10 monclient(hunting): dump: 2024-03-25T19:27:38.761 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:38.761 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:38.761 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:38.761 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:38.762 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:38.762 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:38.762 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:38.762 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:38.762 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:38.762 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:38.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.761+0000 7f128ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:38.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1463288332 0 0) 0x7f12a812f780 con 0x7f12a80b1730 2024-03-25T19:27:38.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 >> v1:172.21.15.123:6790/0 conn(0x7f12a814be90 legacy=0x7f12a81379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 >> v1:172.21.15.151:6789/0 conn(0x7f12a80b4d60 legacy=0x7f12a812eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:38.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 1 monclient: found mon.a 2024-03-25T19:27:38.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:38.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:38.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a813dd20 con 0x7f12a80b1730 2024-03-25T19:27:38.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 10 monclient: _finish_auth 0 2024-03-25T19:27:38.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:38.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f128ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:38.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f12adf54640 5 monclient: authenticate success, global_id 4437 2024-03-25T19:27:38.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f12adf54640 10 monclient: _renew_subs 2024-03-25T19:27:38.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f12adf54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:38.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f12adf54640 1 -- 172.21.15.123:0/1369157841 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a813df70 con 0x7f12a80b1730 2024-03-25T19:27:38.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f12adf54640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:38.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f12adf54640 10 monclient: _renew_subs 2024-03-25T19:27:38.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f12adf54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:38.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f12adf54640 1 -- 172.21.15.123:0/1369157841 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a8135f50 con 0x7f12a80b1730 2024-03-25T19:27:38.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.844+0000 7f12adf54640 1 librados: init done 2024-03-25T19:27:38.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.845+0000 7f12adf54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:38.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.845+0000 7f12adf54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:38.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.845+0000 7f12adf54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:38.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:38.845+0000 7f12adf54640 1 -- 172.21.15.123:0/1369157841 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a8061ca0 con 0x7f12a80b1730 2024-03-25T19:27:39.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.106+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1298001210 con 0x7f12a80b1730 2024-03-25T19:27:39.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.106+0000 7f128ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:39.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.106+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1298004c30 con 0x7f12a80b1730 2024-03-25T19:27:39.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.106+0000 7f128ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:39.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.106+0000 7f128ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:39.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.106+0000 7f128ffff640 10 monclient: dump: 2024-03-25T19:27:39.106 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:39.106 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:39.106 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:39.107 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:39.107 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:39.107 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:39.107 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:39.107 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:39.107 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:39.107 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:39.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.106+0000 7f12a5ffb640 4 set_mon_vals no callback set 2024-03-25T19:27:39.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.106+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f129801ac00 con 0x7f12a80b1730 2024-03-25T19:27:39.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.107+0000 7f12a6ffd640 10 monclient: get_auth_request con 0x7f129406de20 auth_method 0 2024-03-25T19:27:39.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.107+0000 7f128ffff640 10 client.4437.objecter ms_handle_connect 0x7f129406de20 2024-03-25T19:27:39.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.111+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f1298054830 con 0x7f12a80b1730 2024-03-25T19:27:39.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.111+0000 7f128ffff640 10 client.4437.objecter ms_dispatch 0x7f12a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:39.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.111+0000 7f128ffff640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:39.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.111+0000 7f128ffff640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:39.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.111+0000 7f128ffff640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T19:27:39.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.111+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1298050170 con 0x7f12a80b1730 2024-03-25T19:27:39.112 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.111+0000 7f128ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:39.112 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.111+0000 7f128ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:39.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.330+0000 7f12adf54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:39.331 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.330+0000 7f12adf54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:39.331 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.330+0000 7f12adf54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:39.331 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.330+0000 7f12adf54640 1 -- 172.21.15.123:0/1369157841 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12a80627b0 con 0x7f12a80b1730 2024-03-25T19:27:39.385 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.385+0000 7f128ffff640 1 -- 172.21.15.123:0/1369157841 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f1298052270 con 0x7f12a80b1730 2024-03-25T19:27:39.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.385+0000 7f128ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:39.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.385+0000 7f128ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:39.386 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:27:39.386 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:39.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 10 librados: watch_flush enter 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 10 librados: watch_flush exit 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 10 client.4437.objecter successfully canceled tick 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 1 -- 172.21.15.123:0/1369157841 >> v1:172.21.15.151:6800/727684239 conn(0x7f129406de20 legacy=0x7f12940702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 10 monclient: shutdown 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 1 -- 172.21.15.123:0/1369157841 >> v1:172.21.15.123:6789/0 conn(0x7f12a80b1730 legacy=0x7f12a8145330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 1 -- 172.21.15.123:0/1369157841 shutdown_connections 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 1 -- 172.21.15.123:0/1369157841 >> 172.21.15.123:0/1369157841 conn(0x7f12a80617a0 msgr2=0x7f12a80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 1 -- 172.21.15.123:0/1369157841 shutdown_connections 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 1 -- 172.21.15.123:0/1369157841 wait complete. 2024-03-25T19:27:39.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:39.388+0000 7f12adf54640 1 librados: shutdown 2024-03-25T19:27:39.401 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:46.206 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:46.326 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.326+0000 7ff415c4e640 10 monclient: get_monmap_and_config 2024-03-25T19:27:46.326 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.326+0000 7ff415c4e640 10 monclient: build_initial_monmap 2024-03-25T19:27:46.326 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.326+0000 7ff415c4e640 10 monclient: monmap: 2024-03-25T19:27:46.326 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:46.326 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:46.326 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:27:46.327254+0000 2024-03-25T19:27:46.326 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:27:46.327254+0000 2024-03-25T19:27:46.327 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:46.327 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:46.327 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:46.327 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:27:46.327 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:27:46.327 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:46.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 1 Processor -- start 2024-03-25T19:27:46.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 1 -- start start 2024-03-25T19:27:46.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 10 monclient: init 2024-03-25T19:27:46.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:46.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:46.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 10 monclient(hunting): picked mon.noname-a con 0x7ff4100b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:46.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 10 monclient(hunting): picked mon.noname-b con 0x7ff4100b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:46.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 10 monclient(hunting): picked mon.noname-c con 0x7ff41014be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:46.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4100af940 con 0x7ff4100b4d60 2024-03-25T19:27:46.328 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4100b0a70 con 0x7ff41014be90 2024-03-25T19:27:46.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4100b0d50 con 0x7ff4100b1730 2024-03-25T19:27:46.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:46.329 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.327+0000 7ff415c4e640 10 monclient(hunting): authenticate will time out at 5903.687858s 2024-03-25T19:27:46.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.330+0000 7ff40effd640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7ff4100b1730 0x7ff4100b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:37772/0 (socket says 172.21.15.123:37772) 2024-03-25T19:27:46.330 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.330+0000 7ff40effd640 1 -- 172.21.15.123:0/2689643846 learned_addr learned my addr 172.21.15.123:0/2689643846 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:46.342 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.342+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510826343 0 0) 0x7ff4100b0d50 con 0x7ff4100b1730 2024-03-25T19:27:46.342 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.342+0000 7ff40e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:46.342 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.342+0000 7ff40e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:46.342 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.342+0000 7ff40e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:46.342 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.342+0000 7ff40e7fc640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:27:46.342 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.342+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f8003520 con 0x7ff4100b1730 2024-03-25T19:27:46.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.345+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921404339 0 0) 0x7ff4100af940 con 0x7ff4100b4d60 2024-03-25T19:27:46.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.345+0000 7ff40e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:46.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.345+0000 7ff40e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:46.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.345+0000 7ff40e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:46.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.345+0000 7ff40e7fc640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:27:46.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.345+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f8003840 con 0x7ff4100b4d60 2024-03-25T19:27:46.347 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 424453159 0 0) 0x7ff3f8003520 con 0x7ff4100b1730 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 >> v1:172.21.15.151:6789/0 conn(0x7ff41014be90 legacy=0x7ff41014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 >> v1:172.21.15.123:6789/0 conn(0x7ff4100b4d60 legacy=0x7ff41013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4100b1030 con 0x7ff4100b1730 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff404001670 con 0x7ff4100b1730 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:46.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff40e7fc640 10 monclient: dump: 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff415c4e640 5 monclient: authenticate success, global_id 4469 2024-03-25T19:27:46.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.347+0000 7ff415c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:46.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.357+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4040017d0 con 0x7ff4100b1730 2024-03-25T19:27:46.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff40e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff40e7fc640 1 -- 172.21.15.123:0/2689643846 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff404001be0 con 0x7ff4100b1730 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff415c4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff415c4e640 4 set_mon_vals no callback set 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff40e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff40e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff40e7fc640 10 monclient: dump: 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:46.358 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:46.359 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:46.359 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:46.359 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:46.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff415c4e640 10 monclient: shutdown 2024-03-25T19:27:46.360 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff415c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff415c4e640 1 -- 172.21.15.123:0/2689643846 >> v1:172.21.15.123:6790/0 conn(0x7ff4100b1730 legacy=0x7ff4100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff415c4e640 1 -- 172.21.15.123:0/2689643846 shutdown_connections 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff415c4e640 1 -- 172.21.15.123:0/2689643846 >> 172.21.15.123:0/2689643846 conn(0x7ff4100617a0 msgr2=0x7ff410061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff415c4e640 1 -- 172.21.15.123:0/2689643846 shutdown_connections 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.358+0000 7ff415c4e640 1 -- 172.21.15.123:0/2689643846 wait complete. 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.359+0000 7ff415c4e640 10 monclient: build_initial_monmap 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.359+0000 7ff415c4e640 10 monclient: monmap: 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:46.361 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.359+0000 7ff415c4e640 1 librados: starting msgr at 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.359+0000 7ff415c4e640 1 librados: starting objecter 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.359+0000 7ff415c4e640 1 Processor -- start 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 1 -- start start 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 1 librados: setting wanted keys 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 1 librados: calling monclient init 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 10 monclient: init 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 10 monclient(hunting): picked mon.noname-c con 0x7ff4100b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:46.362 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 10 monclient(hunting): picked mon.noname-a con 0x7ff4100b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:46.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 10 monclient(hunting): picked mon.noname-b con 0x7ff41014be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:46.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41013a6d0 con 0x7ff4100b4d60 2024-03-25T19:27:46.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff410139360 con 0x7ff41014be90 2024-03-25T19:27:46.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff410138e40 con 0x7ff4100b1730 2024-03-25T19:27:46.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:46.363 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.360+0000 7ff415c4e640 10 monclient(hunting): authenticate will time out at 5903.720731s 2024-03-25T19:27:46.372 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.372+0000 7ff40f7fe640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7ff4100b1730 0x7ff410131d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:37792/0 (socket says 172.21.15.123:37792) 2024-03-25T19:27:46.372 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.372+0000 7ff40f7fe640 1 -- 172.21.15.123:0/2867980920 learned_addr learned my addr 172.21.15.123:0/2867980920 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:46.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.374+0000 7ff3f7fff640 10 client.?.objecter ms_handle_connect 0x7ff41014be90 2024-03-25T19:27:46.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.374+0000 7ff3f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:46.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.376+0000 7ff3f7fff640 10 client.?.objecter ms_handle_connect 0x7ff4100b1730 2024-03-25T19:27:46.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.376+0000 7ff3f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:46.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.378+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465110264 0 0) 0x7ff410139360 con 0x7ff41014be90 2024-03-25T19:27:46.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.378+0000 7ff3f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:46.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.378+0000 7ff3f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:46.378 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.378+0000 7ff3f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:46.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.378+0000 7ff3f7fff640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:27:46.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.378+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc004eb0 con 0x7ff41014be90 2024-03-25T19:27:46.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.381+0000 7ff3f7fff640 10 client.?.objecter ms_handle_connect 0x7ff4100b4d60 2024-03-25T19:27:46.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.381+0000 7ff3f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:46.383 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.384+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246040127 0 0) 0x7ff410138e40 con 0x7ff4100b1730 2024-03-25T19:27:46.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.384+0000 7ff3f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:46.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.384+0000 7ff3f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:46.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.384+0000 7ff3f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:46.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.384+0000 7ff3f7fff640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:27:46.384 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.384+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff410139360 con 0x7ff4100b1730 2024-03-25T19:27:46.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.391+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 83135442 0 0) 0x7ff3fc004eb0 con 0x7ff41014be90 2024-03-25T19:27:46.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.391+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff410138e40 con 0x7ff41014be90 2024-03-25T19:27:46.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.391+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff408001be0 con 0x7ff41014be90 2024-03-25T19:27:46.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.391+0000 7ff3f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:46.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.391+0000 7ff3f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:46.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.391+0000 7ff3f7fff640 10 monclient(hunting): dump: 2024-03-25T19:27:46.391 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:46.391 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:46.391 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:46.392 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:46.392 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:46.392 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:46.392 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:46.392 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:46.392 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:46.392 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:46.392 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.391+0000 7ff3f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:46.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667755357 0 0) 0x7ff41013a6d0 con 0x7ff4100b4d60 2024-03-25T19:27:46.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:46.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:46.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:46.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:27:46.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc004eb0 con 0x7ff4100b4d60 2024-03-25T19:27:46.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2459878920 0 0) 0x7ff410139360 con 0x7ff4100b1730 2024-03-25T19:27:46.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff41013a6d0 con 0x7ff4100b1730 2024-03-25T19:27:46.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff4000019f0 con 0x7ff4100b1730 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.392+0000 7ff3f7fff640 10 monclient(hunting): dump: 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:46.394 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:46.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 471202494 0 0) 0x7ff41013a6d0 con 0x7ff4100b1730 2024-03-25T19:27:46.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 >> v1:172.21.15.151:6789/0 conn(0x7ff41014be90 legacy=0x7ff41013b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 >> v1:172.21.15.123:6789/0 conn(0x7ff4100b4d60 legacy=0x7ff41013ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:46.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 1 monclient: found mon.c 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410138fd0 con 0x7ff4100b1730 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff3f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff415c4e640 5 monclient: authenticate success, global_id 4472 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff415c4e640 10 monclient: _renew_subs 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff415c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff415c4e640 1 -- 172.21.15.123:0/2867980920 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4101386b0 con 0x7ff4100b1730 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff415c4e640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff415c4e640 10 monclient: _renew_subs 2024-03-25T19:27:46.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff415c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:27:46.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff415c4e640 1 -- 172.21.15.123:0/2867980920 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff410145650 con 0x7ff4100b1730 2024-03-25T19:27:46.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.401+0000 7ff415c4e640 1 librados: init done 2024-03-25T19:27:46.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.402+0000 7ff415c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:46.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.402+0000 7ff415c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:46.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.402+0000 7ff415c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:27:46.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.402+0000 7ff415c4e640 1 -- 172.21.15.123:0/2867980920 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4101391c0 con 0x7ff4100b1730 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.713+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff400003610 con 0x7ff4100b1730 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.713+0000 7ff3f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.713+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ff400003880 con 0x7ff4100b1730 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.713+0000 7ff3f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.713+0000 7ff3f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.713+0000 7ff3f7fff640 10 monclient: dump: 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:46.713 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:46.714 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:46.714 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:46.714 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:46.714 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:46.714 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.713+0000 7ff40dffb640 4 set_mon_vals no callback set 2024-03-25T19:27:46.714 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.713+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7ff400003aa0 con 0x7ff4100b1730 2024-03-25T19:27:46.715 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.714+0000 7ff40effd640 10 monclient: get_auth_request con 0x7ff3fc06df00 auth_method 0 2024-03-25T19:27:46.715 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.714+0000 7ff3f7fff640 10 client.4472.objecter ms_handle_connect 0x7ff3fc06df00 2024-03-25T19:27:46.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.719+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7ff400054680 con 0x7ff4100b1730 2024-03-25T19:27:46.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.719+0000 7ff3f7fff640 10 client.4472.objecter ms_dispatch 0x7ff410000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:46.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.719+0000 7ff3f7fff640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:46.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.719+0000 7ff3f7fff640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:46.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.719+0000 7ff3f7fff640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-25T19:27:46.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.719+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff400050190 con 0x7ff4100b1730 2024-03-25T19:27:46.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.719+0000 7ff3f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:46.719 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.719+0000 7ff3f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:46.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.946+0000 7ff415c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:46.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.946+0000 7ff415c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:46.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.946+0000 7ff415c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:27:46.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:46.946+0000 7ff415c4e640 1 -- 172.21.15.123:0/2867980920 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff410138a50 con 0x7ff4100b1730 2024-03-25T19:27:47.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.115+0000 7ff3f7fff640 1 -- 172.21.15.123:0/2867980920 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7ff4000520c0 con 0x7ff4100b1730 2024-03-25T19:27:47.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.115+0000 7ff3f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:47.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.115+0000 7ff3f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:47.116 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:27:47.116 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.117+0000 7ff415c4e640 10 librados: watch_flush enter 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.117+0000 7ff415c4e640 10 librados: watch_flush exit 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.117+0000 7ff415c4e640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.117+0000 7ff415c4e640 10 client.4472.objecter successfully canceled tick 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.117+0000 7ff415c4e640 1 -- 172.21.15.123:0/2867980920 >> v1:172.21.15.151:6800/727684239 conn(0x7ff3fc06df00 legacy=0x7ff3fc0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.117+0000 7ff415c4e640 10 monclient: shutdown 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.117+0000 7ff415c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.117+0000 7ff415c4e640 1 -- 172.21.15.123:0/2867980920 >> v1:172.21.15.123:6790/0 conn(0x7ff4100b1730 legacy=0x7ff410131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.118+0000 7ff415c4e640 1 -- 172.21.15.123:0/2867980920 shutdown_connections 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.118+0000 7ff415c4e640 1 -- 172.21.15.123:0/2867980920 >> 172.21.15.123:0/2867980920 conn(0x7ff4100617a0 msgr2=0x7ff41013c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.118+0000 7ff415c4e640 1 -- 172.21.15.123:0/2867980920 shutdown_connections 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.118+0000 7ff415c4e640 1 -- 172.21.15.123:0/2867980920 wait complete. 2024-03-25T19:27:47.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:47.118+0000 7ff415c4e640 1 librados: shutdown 2024-03-25T19:27:47.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:50.629 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:05:40.201 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:50.633 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:05:40.205 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:50.680 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:05:40.252 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:50.682 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:05:40.216 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:50.692 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:05:40.226 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:50.694 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:05:40.266 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:50.700 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:05:40.272 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:50.709 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:05:40.243 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:50.729 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:05:40.301 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:53.937 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.053+0000 7f4e90110640 10 monclient: get_monmap_and_config 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.053+0000 7f4e90110640 10 monclient: build_initial_monmap 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.053+0000 7f4e90110640 10 monclient: monmap: 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:27:54.054021+0000 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:27:54.054021+0000 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:54.053 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:27:54.054 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:27:54.054 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:54.054 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.053+0000 7f4e90110640 1 Processor -- start 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.053+0000 7f4e90110640 1 -- start start 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 10 monclient: init 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 10 monclient(hunting): picked mon.noname-c con 0x7f4e880b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 10 monclient(hunting): picked mon.noname-a con 0x7f4e880b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 10 monclient(hunting): picked mon.noname-b con 0x7f4e8814be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e880af940 con 0x7f4e880b1730 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e880b0a70 con 0x7f4e880b4d60 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e880b0d50 con 0x7f4e8814be90 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:54.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.054+0000 7f4e90110640 10 monclient(hunting): authenticate will time out at 5911.414680s 2024-03-25T19:27:54.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.063+0000 7f4e8deac640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4e880b4d60 0x7f4e8813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:45144/0 (socket says 172.21.15.123:45144) 2024-03-25T19:27:54.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.063+0000 7f4e8deac640 1 -- 172.21.15.123:0/2761084282 learned_addr learned my addr 172.21.15.123:0/2761084282 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:54.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.078+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12391965 0 0) 0x7f4e880b0a70 con 0x7f4e880b4d60 2024-03-25T19:27:54.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.078+0000 7f4e8ceaa640 10 monclient(hunting): _negotiate 2024-03-25T19:27:54.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.078+0000 7f4e8ceaa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:54.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.078+0000 7f4e8ceaa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:54.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.078+0000 7f4e8ceaa640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:27:54.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.078+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e74003520 con 0x7f4e880b4d60 2024-03-25T19:27:54.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.083+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332696439 0 0) 0x7f4e880b0d50 con 0x7f4e8814be90 2024-03-25T19:27:54.084 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.083+0000 7f4e8ceaa640 10 monclient(hunting): _negotiate 2024-03-25T19:27:54.084 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.083+0000 7f4e8ceaa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:54.084 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.083+0000 7f4e8ceaa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:54.084 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.083+0000 7f4e8ceaa640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:27:54.084 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.084+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e880b0a70 con 0x7f4e8814be90 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665710918 0 0) 0x7f4e74003520 con 0x7f4e880b4d60 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 >> v1:172.21.15.123:6790/0 conn(0x7f4e8814be90 legacy=0x7f4e8814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 >> v1:172.21.15.123:6789/0 conn(0x7f4e880b1730 legacy=0x7f4e880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 10 monclient: _finish_hunting 0 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 1 monclient: found mon.noname-c 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e880b1030 con 0x7f4e880b4d60 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 10 monclient: _finish_auth 0 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 10 monclient: _check_auth_tickets 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:54.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4e84001690 con 0x7f4e880b4d60 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e8ceaa640 10 monclient: dump: 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e90110640 5 monclient: authenticate success, global_id 4453 2024-03-25T19:27:54.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.088+0000 7f4e90110640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.100+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e840017f0 con 0x7f4e880b4d60 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.100+0000 7f4e8ceaa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.100+0000 7f4e8ceaa640 1 -- 172.21.15.123:0/2761084282 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4e84001c00 con 0x7f4e880b4d60 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.100+0000 7f4e90110640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.100+0000 7f4e90110640 4 set_mon_vals no callback set 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.100+0000 7f4e8ceaa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.100+0000 7f4e8ceaa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.100+0000 7f4e8ceaa640 10 monclient: dump: 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:54.100 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.101 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.101 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:54.101 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:54.101 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:54.101 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:54.101 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:54.101 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:54.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.101+0000 7f4e90110640 10 monclient: shutdown 2024-03-25T19:27:54.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.101+0000 7f4e90110640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:54.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.101+0000 7f4e90110640 1 -- 172.21.15.123:0/2761084282 >> v1:172.21.15.151:6789/0 conn(0x7f4e880b4d60 legacy=0x7f4e8813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.101+0000 7f4e90110640 1 -- 172.21.15.123:0/2761084282 shutdown_connections 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.101+0000 7f4e90110640 1 -- 172.21.15.123:0/2761084282 >> 172.21.15.123:0/2761084282 conn(0x7f4e880617a0 msgr2=0x7f4e88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.101+0000 7f4e90110640 1 -- 172.21.15.123:0/2761084282 shutdown_connections 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.101+0000 7f4e90110640 1 -- 172.21.15.123:0/2761084282 wait complete. 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.102+0000 7f4e90110640 10 monclient: build_initial_monmap 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.102+0000 7f4e90110640 10 monclient: monmap: 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:27:54.103 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:27:54.104 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:27:54.104 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:27:54.104 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:54.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.102+0000 7f4e90110640 1 librados: starting msgr at 2024-03-25T19:27:54.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.102+0000 7f4e90110640 1 librados: starting objecter 2024-03-25T19:27:54.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.102+0000 7f4e90110640 1 Processor -- start 2024-03-25T19:27:54.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.102+0000 7f4e90110640 1 -- start start 2024-03-25T19:27:54.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.102+0000 7f4e90110640 1 librados: setting wanted keys 2024-03-25T19:27:54.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.102+0000 7f4e90110640 1 librados: calling monclient init 2024-03-25T19:27:54.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.102+0000 7f4e90110640 10 monclient: init 2024-03-25T19:27:54.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.103+0000 7f4e90110640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:54.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.103+0000 7f4e90110640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:27:54.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.103+0000 7f4e90110640 10 monclient(hunting): picked mon.noname-b con 0x7f4e880b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:27:54.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.103+0000 7f4e90110640 10 monclient(hunting): picked mon.noname-a con 0x7f4e880b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:27:54.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.103+0000 7f4e90110640 10 monclient(hunting): picked mon.noname-c con 0x7f4e8814be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:27:54.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.103+0000 7f4e90110640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e88137070 con 0x7f4e880b4d60 2024-03-25T19:27:54.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.103+0000 7f4e90110640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e881375c0 con 0x7f4e880b1730 2024-03-25T19:27:54.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.103+0000 7f4e90110640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e881378a0 con 0x7f4e8814be90 2024-03-25T19:27:54.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.103+0000 7f4e90110640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:54.106 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.103+0000 7f4e90110640 10 monclient(hunting): authenticate will time out at 5911.463398s 2024-03-25T19:27:54.113 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.113+0000 7f4e8d6ab640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4e880b4d60 0x7f4e88133be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:56038/0 (socket says 172.21.15.123:56038) 2024-03-25T19:27:54.113 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.113+0000 7f4e8d6ab640 1 -- 172.21.15.123:0/2569664854 learned_addr learned my addr 172.21.15.123:0/2569664854 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:27:54.113 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.113+0000 7f4e7e7fc640 10 client.?.objecter ms_handle_connect 0x7f4e880b4d60 2024-03-25T19:27:54.113 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.113+0000 7f4e7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:54.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.119+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438051914 0 0) 0x7f4e88137070 con 0x7f4e880b4d60 2024-03-25T19:27:54.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.119+0000 7f4e7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:54.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.119+0000 7f4e7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:54.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.119+0000 7f4e7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:54.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.119+0000 7f4e7e7fc640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:27:54.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.119+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e6c004eb0 con 0x7f4e880b4d60 2024-03-25T19:27:54.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.119+0000 7f4e7e7fc640 10 client.?.objecter ms_handle_connect 0x7f4e880b1730 2024-03-25T19:27:54.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.119+0000 7f4e7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:54.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.128+0000 7f4e7e7fc640 10 client.?.objecter ms_handle_connect 0x7f4e8814be90 2024-03-25T19:27:54.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.128+0000 7f4e7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:54.132 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.132+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182311663 0 0) 0x7f4e881375c0 con 0x7f4e880b1730 2024-03-25T19:27:54.132 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.132+0000 7f4e7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:54.133 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.132+0000 7f4e7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:54.133 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.132+0000 7f4e7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:54.133 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.132+0000 7f4e7e7fc640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:27:54.133 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.133+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e88137070 con 0x7f4e880b1730 2024-03-25T19:27:54.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.141+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895777420 0 0) 0x7f4e881378a0 con 0x7f4e8814be90 2024-03-25T19:27:54.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.141+0000 7f4e7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:54.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.141+0000 7f4e7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:54.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.141+0000 7f4e7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:54.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.141+0000 7f4e7e7fc640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:27:54.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.141+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e881375c0 con 0x7f4e8814be90 2024-03-25T19:27:54.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.143+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 532348001 0 0) 0x7f4e6c004eb0 con 0x7f4e880b4d60 2024-03-25T19:27:54.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.143+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e881378a0 con 0x7f4e880b4d60 2024-03-25T19:27:54.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.143+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4e780019d0 con 0x7f4e880b4d60 2024-03-25T19:27:54.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.143+0000 7f4e7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:54.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.143+0000 7f4e7e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:54.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.143+0000 7f4e7e7fc640 10 monclient(hunting): dump: 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:54.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.143+0000 7f4e7e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:54.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.151+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3043305900 0 0) 0x7f4e88137070 con 0x7f4e880b1730 2024-03-25T19:27:54.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.151+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e6c004eb0 con 0x7f4e880b1730 2024-03-25T19:27:54.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.151+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4e84001b00 con 0x7f4e880b1730 2024-03-25T19:27:54.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.151+0000 7f4e7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:54.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.151+0000 7f4e7e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:54.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.151+0000 7f4e7e7fc640 10 monclient(hunting): dump: 2024-03-25T19:27:54.151 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:54.151 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:54.151 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.152 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.152 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:54.152 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:54.152 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:54.152 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:54.152 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:54.152 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:54.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.159+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669280185 0 0) 0x7f4e881375c0 con 0x7f4e8814be90 2024-03-25T19:27:54.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.159+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e88137070 con 0x7f4e8814be90 2024-03-25T19:27:54.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.159+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4e80001bd0 con 0x7f4e8814be90 2024-03-25T19:27:54.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.159+0000 7f4e7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:54.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.159+0000 7f4e7e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:54.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.159+0000 7f4e7e7fc640 10 monclient(hunting): dump: 2024-03-25T19:27:54.162 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:54.162 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:54.162 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.162 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.163 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:54.163 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:54.163 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:54.163 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:54.163 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:54.163 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:54.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4287887628 0 0) 0x7f4e881378a0 con 0x7f4e880b4d60 2024-03-25T19:27:54.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 >> v1:172.21.15.123:6790/0 conn(0x7f4e8814be90 legacy=0x7f4e881342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:54.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 >> v1:172.21.15.151:6789/0 conn(0x7f4e880b1730 legacy=0x7f4e88137cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:54.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:54.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 1 monclient: found mon.a 2024-03-25T19:27:54.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:54.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:54.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e88134a00 con 0x7f4e880b4d60 2024-03-25T19:27:54.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:54.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:54.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:54.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e90110640 5 monclient: authenticate success, global_id 4449 2024-03-25T19:27:54.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e90110640 10 monclient: _renew_subs 2024-03-25T19:27:54.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e90110640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:54.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e90110640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e881372e0 con 0x7f4e880b4d60 2024-03-25T19:27:54.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e90110640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:54.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e90110640 10 monclient: _renew_subs 2024-03-25T19:27:54.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e90110640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:54.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e90110640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e881365f0 con 0x7f4e880b4d60 2024-03-25T19:27:54.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.163+0000 7f4e90110640 1 librados: init done 2024-03-25T19:27:54.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.164+0000 7f4e90110640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:54.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.164+0000 7f4e90110640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:54.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.164+0000 7f4e90110640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:54.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.164+0000 7f4e90110640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e880627b0 con 0x7f4e880b4d60 2024-03-25T19:27:54.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.484+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e78003610 con 0x7f4e880b4d60 2024-03-25T19:27:54.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.484+0000 7f4e7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:54.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.484+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4e78003a20 con 0x7f4e880b4d60 2024-03-25T19:27:54.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.484+0000 7f4e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:54.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.484+0000 7f4e7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.484+0000 7f4e7e7fc640 10 monclient: dump: 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:27:54.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.484+0000 7f4e7ffff640 4 set_mon_vals no callback set 2024-03-25T19:27:54.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.485+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f4e78003c40 con 0x7f4e880b4d60 2024-03-25T19:27:54.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.485+0000 7f4e8deac640 10 monclient: get_auth_request con 0x7f4e6c06e0d0 auth_method 0 2024-03-25T19:27:54.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.485+0000 7f4e7e7fc640 10 client.4449.objecter ms_handle_connect 0x7f4e6c06e0d0 2024-03-25T19:27:54.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.490+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f4e780547d0 con 0x7f4e880b4d60 2024-03-25T19:27:54.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.490+0000 7f4e7e7fc640 10 client.4449.objecter ms_dispatch 0x7f4e88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:54.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.490+0000 7f4e7e7fc640 3 client.4449.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:54.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.490+0000 7f4e7e7fc640 3 client.4449.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:54.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.491+0000 7f4e7e7fc640 20 client.4449.objecter dump_active .. 0 homeless 2024-03-25T19:27:54.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.491+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e780502e0 con 0x7f4e880b4d60 2024-03-25T19:27:54.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.491+0000 7f4e7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:54.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.491+0000 7f4e7e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:54.739 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.739+0000 7f4e90110640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:54.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.739+0000 7f4e90110640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:54.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.739+0000 7f4e90110640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:27:54.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.739+0000 7f4e90110640 1 -- 172.21.15.123:0/2569664854 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e88069f40 con 0x7f4e880b4d60 2024-03-25T19:27:54.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.793+0000 7f4e7e7fc640 1 -- 172.21.15.123:0/2569664854 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f4e78052210 con 0x7f4e880b4d60 2024-03-25T19:27:54.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.793+0000 7f4e7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:54.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.793+0000 7f4e7e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:54.794 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:27:54.794 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:54.796 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 10 librados: watch_flush enter 2024-03-25T19:27:54.796 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 10 librados: watch_flush exit 2024-03-25T19:27:54.796 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 20 client.4449.objecter shutdown clearing up homeless session... 2024-03-25T19:27:54.796 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 10 client.4449.objecter successfully canceled tick 2024-03-25T19:27:54.796 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 1 -- 172.21.15.123:0/2569664854 >> v1:172.21.15.151:6800/727684239 conn(0x7f4e6c06e0d0 legacy=0x7f4e6c070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:54.796 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 10 monclient: shutdown 2024-03-25T19:27:54.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:54.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 1 -- 172.21.15.123:0/2569664854 >> v1:172.21.15.123:6789/0 conn(0x7f4e880b4d60 legacy=0x7f4e88133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:54.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 1 -- 172.21.15.123:0/2569664854 shutdown_connections 2024-03-25T19:27:54.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 1 -- 172.21.15.123:0/2569664854 >> 172.21.15.123:0/2569664854 conn(0x7f4e880617a0 msgr2=0x7f4e88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:54.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 1 -- 172.21.15.123:0/2569664854 shutdown_connections 2024-03-25T19:27:54.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 1 -- 172.21.15.123:0/2569664854 wait complete. 2024-03-25T19:27:54.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:27:54.796+0000 7f4e90110640 1 librados: shutdown 2024-03-25T19:27:54.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:01.612 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.729+0000 7f3488123640 10 monclient: get_monmap_and_config 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.729+0000 7f3488123640 10 monclient: build_initial_monmap 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.729+0000 7f3488123640 10 monclient: monmap: 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:28:01.731607+0000 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:28:01.731607+0000 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:01.731 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:28:01.732 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:28:01.732 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:01.732 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.730+0000 7f3488123640 1 Processor -- start 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.730+0000 7f3488123640 1 -- start start 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.730+0000 7f3488123640 10 monclient: init 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.731+0000 7f3488123640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.731+0000 7f3488123640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.731+0000 7f3488123640 10 monclient(hunting): picked mon.noname-c con 0x7f34800a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.731+0000 7f3488123640 10 monclient(hunting): picked mon.noname-a con 0x7f348013f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.731+0000 7f3488123640 10 monclient(hunting): picked mon.noname-b con 0x7f34801430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.731+0000 7f3488123640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3480142520 con 0x7f348013f710 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.731+0000 7f3488123640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34800b28a0 con 0x7f34800a98f0 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.731+0000 7f3488123640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34800b2a30 con 0x7f34801430c0 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.731+0000 7f3488123640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:01.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.731+0000 7f3488123640 10 monclient(hunting): authenticate will time out at 5919.092515s 2024-03-25T19:28:01.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.746+0000 7f34856be640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f348013f710 0x7f34801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:59290/0 (socket says 172.21.15.123:59290) 2024-03-25T19:28:01.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.746+0000 7f34856be640 1 -- 172.21.15.123:0/2077957157 learned_addr learned my addr 172.21.15.123:0/2077957157 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:01.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.763+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198907795 0 0) 0x7f34800b2a30 con 0x7f34801430c0 2024-03-25T19:28:01.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.763+0000 7f3484ebd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.763+0000 7f3484ebd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.763+0000 7f3484ebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.763+0000 7f3484ebd640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:28:01.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.763+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f346c003520 con 0x7f34801430c0 2024-03-25T19:28:01.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.769+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924927100 0 0) 0x7f3480142520 con 0x7f348013f710 2024-03-25T19:28:01.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.769+0000 7f3484ebd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.769+0000 7f3484ebd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.769+0000 7f3484ebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.769+0000 7f3484ebd640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:28:01.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.769+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34800b2a30 con 0x7f348013f710 2024-03-25T19:28:01.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.773+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347996924 0 0) 0x7f34800b28a0 con 0x7f34800a98f0 2024-03-25T19:28:01.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.773+0000 7f3484ebd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.773+0000 7f3484ebd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.773+0000 7f3484ebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.773+0000 7f3484ebd640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:28:01.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.773+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3480142520 con 0x7f34800a98f0 2024-03-25T19:28:01.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2973168354 0 0) 0x7f346c003520 con 0x7f34801430c0 2024-03-25T19:28:01.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 >> v1:172.21.15.151:6789/0 conn(0x7f34800a98f0 legacy=0x7f34800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 >> v1:172.21.15.123:6789/0 conn(0x7f348013f710 legacy=0x7f34801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:01.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 1 monclient: found mon.noname-b 2024-03-25T19:28:01.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:01.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34800b44c0 con 0x7f34801430c0 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 10 monclient: _finish_auth 0 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f3478001a10 con 0x7f34801430c0 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3484ebd640 10 monclient: dump: 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:01.790 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:01.791 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:01.791 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:01.791 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:01.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3488123640 5 monclient: authenticate success, global_id 4481 2024-03-25T19:28:01.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.788+0000 7f3488123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:01.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.812+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3478001b70 con 0x7f34801430c0 2024-03-25T19:28:01.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.812+0000 7f3484ebd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:01.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.812+0000 7f3484ebd640 1 -- 172.21.15.123:0/2077957157 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f3478001f80 con 0x7f34801430c0 2024-03-25T19:28:01.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.812+0000 7f3484ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.812+0000 7f3484ebd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.812+0000 7f3484ebd640 10 monclient: dump: 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:01.814 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:01.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.812+0000 7f3488123640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:01.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.812+0000 7f3488123640 4 set_mon_vals no callback set 2024-03-25T19:28:01.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.813+0000 7f3488123640 10 monclient: shutdown 2024-03-25T19:28:01.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.813+0000 7f3488123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:01.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.813+0000 7f3488123640 1 -- 172.21.15.123:0/2077957157 >> v1:172.21.15.123:6790/0 conn(0x7f34801430c0 legacy=0x7f348014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.813+0000 7f3488123640 1 -- 172.21.15.123:0/2077957157 shutdown_connections 2024-03-25T19:28:01.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.813+0000 7f3488123640 1 -- 172.21.15.123:0/2077957157 >> 172.21.15.123:0/2077957157 conn(0x7f34800b7ea0 msgr2=0x7f34800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:01.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.813+0000 7f3488123640 1 -- 172.21.15.123:0/2077957157 shutdown_connections 2024-03-25T19:28:01.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.813+0000 7f3488123640 1 -- 172.21.15.123:0/2077957157 wait complete. 2024-03-25T19:28:01.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.814+0000 7f3488123640 10 monclient: build_initial_monmap 2024-03-25T19:28:01.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.814+0000 7f3488123640 10 monclient: monmap: 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.814+0000 7f3488123640 1 librados: starting msgr at 2024-03-25T19:28:01.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.814+0000 7f3488123640 1 librados: starting objecter 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.814+0000 7f3488123640 1 Processor -- start 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.814+0000 7f3488123640 1 -- start start 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.814+0000 7f3488123640 1 librados: setting wanted keys 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.814+0000 7f3488123640 1 librados: calling monclient init 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.814+0000 7f3488123640 10 monclient: init 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.815+0000 7f3488123640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.815+0000 7f3488123640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.815+0000 7f3488123640 10 monclient(hunting): picked mon.noname-a con 0x7f34800a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.815+0000 7f3488123640 10 monclient(hunting): picked mon.noname-b con 0x7f348013f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.815+0000 7f3488123640 10 monclient(hunting): picked mon.noname-c con 0x7f34801430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.815+0000 7f3488123640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3480137ab0 con 0x7f34800a98f0 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.815+0000 7f3488123640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3480138bd0 con 0x7f348013f710 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.815+0000 7f3488123640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3480138e40 con 0x7f34801430c0 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.815+0000 7f3488123640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:01.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.815+0000 7f3488123640 10 monclient(hunting): authenticate will time out at 5919.176812s 2024-03-25T19:28:01.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.824+0000 7f34856be640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f348013f710 0x7f34801df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:55324/0 (socket says 172.21.15.123:55324) 2024-03-25T19:28:01.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.824+0000 7f34856be640 1 -- 172.21.15.123:0/1741183406 learned_addr learned my addr 172.21.15.123:0/1741183406 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:01.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.830+0000 7f34767fc640 10 client.?.objecter ms_handle_connect 0x7f348013f710 2024-03-25T19:28:01.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.830+0000 7f34767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:01.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.834+0000 7f34767fc640 10 client.?.objecter ms_handle_connect 0x7f34801430c0 2024-03-25T19:28:01.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.834+0000 7f34767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:01.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.839+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867674016 0 0) 0x7f3480138e40 con 0x7f34801430c0 2024-03-25T19:28:01.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.839+0000 7f34767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.839+0000 7f34767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.839+0000 7f34767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.839+0000 7f34767fc640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:28:01.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.839+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3464004eb0 con 0x7f34801430c0 2024-03-25T19:28:01.849 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.848+0000 7f34767fc640 10 client.?.objecter ms_handle_connect 0x7f34800a98f0 2024-03-25T19:28:01.849 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.848+0000 7f34767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:01.852 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.851+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177684739 0 0) 0x7f3480138bd0 con 0x7f348013f710 2024-03-25T19:28:01.852 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.851+0000 7f34767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.852 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.851+0000 7f34767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.852 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.851+0000 7f34767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.852 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.851+0000 7f34767fc640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:28:01.852 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.851+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3480138e40 con 0x7f348013f710 2024-03-25T19:28:01.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.861+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829374011 0 0) 0x7f3480137ab0 con 0x7f34800a98f0 2024-03-25T19:28:01.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.861+0000 7f34767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.861+0000 7f34767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.861+0000 7f34767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.861+0000 7f34767fc640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:28:01.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.861+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3480138bd0 con 0x7f34800a98f0 2024-03-25T19:28:01.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.867+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793240037 0 0) 0x7f3464004eb0 con 0x7f34801430c0 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.867+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3480137ab0 con 0x7f34801430c0 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.867+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f3478001e80 con 0x7f34801430c0 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.867+0000 7f34767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.867+0000 7f34767fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.867+0000 7f34767fc640 10 monclient(hunting): dump: 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:01.869 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:01.870 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:01.870 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:01.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.867+0000 7f34767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:01.875 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.874+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1559369947 0 0) 0x7f3480138bd0 con 0x7f34800a98f0 2024-03-25T19:28:01.875 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.874+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3464004eb0 con 0x7f34800a98f0 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.874+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f347c001830 con 0x7f34800a98f0 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.874+0000 7f34767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.874+0000 7f34767fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.874+0000 7f34767fc640 10 monclient(hunting): dump: 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:01.876 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:01.877 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:01.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.880+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3618620422 0 0) 0x7f3480138e40 con 0x7f348013f710 2024-03-25T19:28:01.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.880+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3480138bd0 con 0x7f348013f710 2024-03-25T19:28:01.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.880+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f3470001970 con 0x7f348013f710 2024-03-25T19:28:01.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.880+0000 7f34767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.880+0000 7f34767fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:01.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.880+0000 7f34767fc640 10 monclient(hunting): dump: 2024-03-25T19:28:01.882 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:01.882 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:01.882 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:01.882 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:01.882 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.882 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:01.882 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:01.882 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:01.882 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:01.882 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:01.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1221902629 0 0) 0x7f3480137ab0 con 0x7f34801430c0 2024-03-25T19:28:01.896 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 >> v1:172.21.15.151:6789/0 conn(0x7f348013f710 legacy=0x7f34801df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.896 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 >> v1:172.21.15.123:6789/0 conn(0x7f34800a98f0 legacy=0x7f3480137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.896 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:01.896 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 1 monclient: found mon.c 2024-03-25T19:28:01.896 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:01.896 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:01.896 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3480139120 con 0x7f34801430c0 2024-03-25T19:28:01.896 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:01.896 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:01.896 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f34767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:01.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f3488123640 5 monclient: authenticate success, global_id 4484 2024-03-25T19:28:01.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f3488123640 10 monclient: _renew_subs 2024-03-25T19:28:01.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f3488123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:01.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f3488123640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3480139400 con 0x7f34801430c0 2024-03-25T19:28:01.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f3488123640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:01.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f3488123640 10 monclient: _renew_subs 2024-03-25T19:28:01.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f3488123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:01.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f3488123640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34800b7880 con 0x7f34801430c0 2024-03-25T19:28:01.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.894+0000 7f3488123640 1 librados: init done 2024-03-25T19:28:01.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.895+0000 7f3488123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:01.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.895+0000 7f3488123640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:01.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.895+0000 7f3488123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:01.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:01.895+0000 7f3488123640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3480137810 con 0x7f34801430c0 2024-03-25T19:28:02.220 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.219+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3478002020 con 0x7f34801430c0 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.219+0000 7f34767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.219+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f3478003dc0 con 0x7f34801430c0 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.219+0000 7f34767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.219+0000 7f34767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.219+0000 7f34767fc640 10 monclient: dump: 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:02.221 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:02.222 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:02.222 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:02.222 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:02.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.219+0000 7f3477fff640 4 set_mon_vals no callback set 2024-03-25T19:28:02.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.220+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f3478019d90 con 0x7f34801430c0 2024-03-25T19:28:02.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.221+0000 7f3485ebf640 10 monclient: get_auth_request con 0x7f346406e0b0 auth_method 0 2024-03-25T19:28:02.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.221+0000 7f34767fc640 10 client.4484.objecter ms_handle_connect 0x7f346406e0b0 2024-03-25T19:28:02.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.225+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f34780538d0 con 0x7f34801430c0 2024-03-25T19:28:02.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.225+0000 7f34767fc640 10 client.4484.objecter ms_dispatch 0x7f3480000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:02.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.225+0000 7f34767fc640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:02.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.225+0000 7f34767fc640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:02.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.226+0000 7f34767fc640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-25T19:28:02.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.226+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f347804f210 con 0x7f34801430c0 2024-03-25T19:28:02.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.226+0000 7f34767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:02.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.226+0000 7f34767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:02.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.470+0000 7f3488123640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:02.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.470+0000 7f3488123640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:02.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.470+0000 7f3488123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:02.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.470+0000 7f3488123640 1 -- 172.21.15.123:0/1741183406 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3480137c60 con 0x7f34801430c0 2024-03-25T19:28:02.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.502+0000 7f34767fc640 1 -- 172.21.15.123:0/1741183406 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f3480137c60 con 0x7f34801430c0 2024-03-25T19:28:02.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.502+0000 7f34767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:02.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.502+0000 7f34767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:02.503 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:28:02.503 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:02.506 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 10 librados: watch_flush enter 2024-03-25T19:28:02.506 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 10 librados: watch_flush exit 2024-03-25T19:28:02.506 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-25T19:28:02.506 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 10 client.4484.objecter successfully canceled tick 2024-03-25T19:28:02.506 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 1 -- 172.21.15.123:0/1741183406 >> v1:172.21.15.151:6800/727684239 conn(0x7f346406e0b0 legacy=0x7f3464070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:02.506 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 10 monclient: shutdown 2024-03-25T19:28:02.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:02.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 1 -- 172.21.15.123:0/1741183406 >> v1:172.21.15.123:6790/0 conn(0x7f34801430c0 legacy=0x7f34801e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:02.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 1 -- 172.21.15.123:0/1741183406 shutdown_connections 2024-03-25T19:28:02.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 1 -- 172.21.15.123:0/1741183406 >> 172.21.15.123:0/1741183406 conn(0x7f34800b7ea0 msgr2=0x7f3480074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:02.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 1 -- 172.21.15.123:0/1741183406 shutdown_connections 2024-03-25T19:28:02.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 1 -- 172.21.15.123:0/1741183406 wait complete. 2024-03-25T19:28:02.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:02.504+0000 7f3488123640 1 librados: shutdown 2024-03-25T19:28:02.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:09.324 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:09.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.440+0000 7fcbc09a8640 10 monclient: get_monmap_and_config 2024-03-25T19:28:09.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.440+0000 7fcbc09a8640 10 monclient: build_initial_monmap 2024-03-25T19:28:09.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.440+0000 7fcbc09a8640 10 monclient: monmap: 2024-03-25T19:28:09.440 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:09.440 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:09.440 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:28:09.441199+0000 2024-03-25T19:28:09.441 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:28:09.441199+0000 2024-03-25T19:28:09.441 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:09.441 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:09.441 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:09.441 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:28:09.441 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:28:09.441 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:09.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.441+0000 7fcbc09a8640 1 Processor -- start 2024-03-25T19:28:09.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.441+0000 7fcbc09a8640 1 -- start start 2024-03-25T19:28:09.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.441+0000 7fcbc09a8640 10 monclient: init 2024-03-25T19:28:09.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.441+0000 7fcbc09a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:09.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.441+0000 7fcbc09a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:09.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.441+0000 7fcbc09a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcbbc0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:09.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.441+0000 7fcbc09a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcbbc0b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:09.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.442+0000 7fcbc09a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcbbc14be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:09.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.442+0000 7fcbc09a8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc0af940 con 0x7fcbbc0b1730 2024-03-25T19:28:09.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.442+0000 7fcbc09a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc0b0a70 con 0x7fcbbc14be90 2024-03-25T19:28:09.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.442+0000 7fcbc09a8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc0b0d50 con 0x7fcbbc0b4d60 2024-03-25T19:28:09.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.442+0000 7fcbc09a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:09.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.442+0000 7fcbc09a8640 10 monclient(hunting): authenticate will time out at 5926.802017s 2024-03-25T19:28:09.447 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.447+0000 7fcbb3fff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fcbbc0b1730 0x7fcbbc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:52356/0 (socket says 172.21.15.123:52356) 2024-03-25T19:28:09.447 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.447+0000 7fcbb3fff640 1 -- 172.21.15.123:0/3794680852 learned_addr learned my addr 172.21.15.123:0/3794680852 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.458+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363095426 0 0) 0x7fcbbc0af940 con 0x7fcbbc0b1730 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.458+0000 7fcbb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.458+0000 7fcbb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.458+0000 7fcbb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.458+0000 7fcbb9d9b640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.458+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba8003520 con 0x7fcbbc0b1730 2024-03-25T19:28:09.465 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.465+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030087556 0 0) 0x7fcbbc0b0d50 con 0x7fcbbc0b4d60 2024-03-25T19:28:09.465 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.465+0000 7fcbb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:09.465 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.465+0000 7fcbb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:09.465 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.465+0000 7fcbb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:09.465 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.465+0000 7fcbb9d9b640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:28:09.465 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.465+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbbc0af940 con 0x7fcbbc0b4d60 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2117401666 0 0) 0x7fcba8003520 con 0x7fcbbc0b1730 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 >> v1:172.21.15.123:6790/0 conn(0x7fcbbc0b4d60 legacy=0x7fcbbc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 >> v1:172.21.15.151:6789/0 conn(0x7fcbbc14be90 legacy=0x7fcbbc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 1 monclient: found mon.noname-a 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0b1030 con 0x7fcbbc0b1730 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fcba4001690 con 0x7fcbbc0b1730 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:09.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbb9d9b640 10 monclient: dump: 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbc09a8640 5 monclient: authenticate success, global_id 4458 2024-03-25T19:28:09.468 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.466+0000 7fcbc09a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:09.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.471+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcba4001870 con 0x7fcbbc0b1730 2024-03-25T19:28:09.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.471+0000 7fcbb9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:09.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.471+0000 7fcbb9d9b640 1 -- 172.21.15.123:0/3794680852 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fcba4001c80 con 0x7fcbbc0b1730 2024-03-25T19:28:09.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.471+0000 7fcbc09a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:09.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.471+0000 7fcbc09a8640 4 set_mon_vals no callback set 2024-03-25T19:28:09.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.471+0000 7fcbb9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:09.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.471+0000 7fcbb9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:09.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.471+0000 7fcbb9d9b640 10 monclient: dump: 2024-03-25T19:28:09.472 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:09.472 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:09.473 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:09.473 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:09.473 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:09.473 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:09.473 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:09.473 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:09.473 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:09.473 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:09.475 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.472+0000 7fcbc09a8640 10 monclient: shutdown 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.472+0000 7fcbc09a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.472+0000 7fcbc09a8640 1 -- 172.21.15.123:0/3794680852 >> v1:172.21.15.123:6789/0 conn(0x7fcbbc0b1730 legacy=0x7fcbbc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.472+0000 7fcbc09a8640 1 -- 172.21.15.123:0/3794680852 shutdown_connections 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.472+0000 7fcbc09a8640 1 -- 172.21.15.123:0/3794680852 >> 172.21.15.123:0/3794680852 conn(0x7fcbbc0617a0 msgr2=0x7fcbbc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.472+0000 7fcbc09a8640 1 -- 172.21.15.123:0/3794680852 shutdown_connections 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.472+0000 7fcbc09a8640 1 -- 172.21.15.123:0/3794680852 wait complete. 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.473+0000 7fcbc09a8640 10 monclient: build_initial_monmap 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.473+0000 7fcbc09a8640 10 monclient: monmap: 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:28:09.476 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 1 librados: starting msgr at 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 1 librados: starting objecter 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 1 Processor -- start 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 1 -- start start 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 1 librados: setting wanted keys 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 1 librados: calling monclient init 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 10 monclient: init 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:09.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:09.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcbbc0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:09.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcbbc0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:09.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcbbc14be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:09.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc1451e0 con 0x7fcbbc14be90 2024-03-25T19:28:09.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc1411f0 con 0x7fcbbc0b1730 2024-03-25T19:28:09.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.474+0000 7fcbc09a8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc141380 con 0x7fcbbc0b4d60 2024-03-25T19:28:09.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.475+0000 7fcbc09a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:09.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.475+0000 7fcbc09a8640 10 monclient(hunting): authenticate will time out at 5926.834899s 2024-03-25T19:28:09.479 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.476+0000 7fcbba59c640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fcbbc0b1730 0x7fcbbc140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:48626/0 (socket says 172.21.15.123:48626) 2024-03-25T19:28:09.479 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.476+0000 7fcbba59c640 1 -- 172.21.15.123:0/2683984389 learned_addr learned my addr 172.21.15.123:0/2683984389 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:09.479 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.477+0000 7fcbb2ffd640 10 client.?.objecter ms_handle_connect 0x7fcbbc0b1730 2024-03-25T19:28:09.479 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.477+0000 7fcbb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:09.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.487+0000 7fcbb2ffd640 10 client.?.objecter ms_handle_connect 0x7fcbbc0b4d60 2024-03-25T19:28:09.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.487+0000 7fcbb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:09.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.490+0000 7fcbb2ffd640 10 client.?.objecter ms_handle_connect 0x7fcbbc14be90 2024-03-25T19:28:09.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.490+0000 7fcbb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:09.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.490+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913119856 0 0) 0x7fcbbc1411f0 con 0x7fcbbc0b1730 2024-03-25T19:28:09.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.490+0000 7fcbb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:09.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.490+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:09.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.490+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:09.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.490+0000 7fcbb2ffd640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:28:09.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.490+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb9c004eb0 con 0x7fcbbc0b1730 2024-03-25T19:28:09.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.496+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731101535 0 0) 0x7fcbbc141380 con 0x7fcbbc0b4d60 2024-03-25T19:28:09.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.496+0000 7fcbb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:09.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.496+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:09.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.496+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:09.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.496+0000 7fcbb2ffd640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:28:09.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.496+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbbc1411f0 con 0x7fcbbc0b4d60 2024-03-25T19:28:09.500 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.500+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791699858 0 0) 0x7fcbbc1451e0 con 0x7fcbbc14be90 2024-03-25T19:28:09.500 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.500+0000 7fcbb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:09.500 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.500+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:09.500 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.500+0000 7fcbb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:09.501 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.500+0000 7fcbb2ffd640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:28:09.501 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.500+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbbc141380 con 0x7fcbbc14be90 2024-03-25T19:28:09.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.503+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1990515427 0 0) 0x7fcb9c004eb0 con 0x7fcbbc0b1730 2024-03-25T19:28:09.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.503+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbbc1451e0 con 0x7fcbbc0b1730 2024-03-25T19:28:09.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.503+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fcbac0019f0 con 0x7fcbbc0b1730 2024-03-25T19:28:09.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.503+0000 7fcbb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:09.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.503+0000 7fcbb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:09.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.503+0000 7fcbb2ffd640 10 monclient(hunting): dump: 2024-03-25T19:28:09.503 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:09.503 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:09.503 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:09.504 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:09.504 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:09.504 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:09.504 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:09.504 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:09.504 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:09.504 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:09.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.503+0000 7fcbb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:09.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 681148573 0 0) 0x7fcbbc1451e0 con 0x7fcbbc0b1730 2024-03-25T19:28:09.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 >> v1:172.21.15.123:6790/0 conn(0x7fcbbc0b4d60 legacy=0x7fcbbc1427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 >> v1:172.21.15.123:6789/0 conn(0x7fcbbc14be90 legacy=0x7fcbbc142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 1 monclient: found mon.b 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc137eb0 con 0x7fcbbc0b1730 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbc09a8640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbc09a8640 10 monclient: _renew_subs 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbc09a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:09.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbc09a8640 1 -- 172.21.15.123:0/2683984389 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbbc138100 con 0x7fcbbc0b1730 2024-03-25T19:28:09.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbc09a8640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:09.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbc09a8640 10 monclient: _renew_subs 2024-03-25T19:28:09.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbc09a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:09.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbc09a8640 1 -- 172.21.15.123:0/2683984389 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbbc1e3710 con 0x7fcbbc0b1730 2024-03-25T19:28:09.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.509+0000 7fcbc09a8640 1 librados: init done 2024-03-25T19:28:09.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.510+0000 7fcbc09a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:09.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.510+0000 7fcbc09a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:09.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.510+0000 7fcbc09a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:09.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.510+0000 7fcbc09a8640 1 -- 172.21.15.123:0/2683984389 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbbc1453d0 con 0x7fcbbc0b1730 2024-03-25T19:28:09.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.823+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbac0035f0 con 0x7fcbbc0b1730 2024-03-25T19:28:09.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.823+0000 7fcbb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:09.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.823+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fcbac003940 con 0x7fcbbc0b1730 2024-03-25T19:28:09.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.823+0000 7fcbb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:09.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.823+0000 7fcbb2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:09.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.823+0000 7fcbb2ffd640 10 monclient: dump: 2024-03-25T19:28:09.824 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:09.824 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:09.824 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:09.824 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:09.824 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:09.825 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:09.825 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:09.825 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:09.825 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:09.825 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:09.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.823+0000 7fcbb959a640 4 set_mon_vals no callback set 2024-03-25T19:28:09.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.824+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fcbac003b60 con 0x7fcbbc0b1730 2024-03-25T19:28:09.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.824+0000 7fcbb3fff640 10 monclient: get_auth_request con 0x7fcb9c06ddf0 auth_method 0 2024-03-25T19:28:09.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.824+0000 7fcbb2ffd640 10 client.4468.objecter ms_handle_connect 0x7fcb9c06ddf0 2024-03-25T19:28:09.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.829+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fcbac054850 con 0x7fcbbc0b1730 2024-03-25T19:28:09.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.829+0000 7fcbb2ffd640 10 client.4468.objecter ms_dispatch 0x7fcbbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:09.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.829+0000 7fcbb2ffd640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:09.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.829+0000 7fcbb2ffd640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:09.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.830+0000 7fcbb2ffd640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T19:28:09.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.830+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbac050360 con 0x7fcbbc0b1730 2024-03-25T19:28:09.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.830+0000 7fcbb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:09.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:09.830+0000 7fcbb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:10.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.058+0000 7fcbc09a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:10.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.058+0000 7fcbc09a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:10.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.058+0000 7fcbc09a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:10.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.058+0000 7fcbc09a8640 1 -- 172.21.15.123:0/2683984389 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbbc0627b0 con 0x7fcbbc0b1730 2024-03-25T19:28:10.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.082+0000 7fcbb2ffd640 1 -- 172.21.15.123:0/2683984389 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fcbac052290 con 0x7fcbbc0b1730 2024-03-25T19:28:10.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.082+0000 7fcbb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:10.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.082+0000 7fcbb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:10.082 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:28:10.082 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:10.084 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 10 librados: watch_flush enter 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 10 librados: watch_flush exit 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 10 client.4468.objecter successfully canceled tick 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 1 -- 172.21.15.123:0/2683984389 >> v1:172.21.15.151:6800/727684239 conn(0x7fcb9c06ddf0 legacy=0x7fcb9c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 10 monclient: shutdown 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 1 -- 172.21.15.123:0/2683984389 >> v1:172.21.15.151:6789/0 conn(0x7fcbbc0b1730 legacy=0x7fcbbc140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 1 -- 172.21.15.123:0/2683984389 shutdown_connections 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 1 -- 172.21.15.123:0/2683984389 >> 172.21.15.123:0/2683984389 conn(0x7fcbbc0617a0 msgr2=0x7fcbbc13c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 1 -- 172.21.15.123:0/2683984389 shutdown_connections 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 1 -- 172.21.15.123:0/2683984389 wait complete. 2024-03-25T19:28:10.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:10.084+0000 7fcbc09a8640 1 librados: shutdown 2024-03-25T19:28:10.096 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:10.699 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:06:00.271 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:10.712 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:06:00.285 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:10.746 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:06:00.318 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:10.762 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:06:00.295 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:10.765 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:06:00.298 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:10.780 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:06:00.352 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:10.783 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:06:00.356 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:10.789 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:06:00.322 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:10.815 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:06:00.387 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:16.903 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.023+0000 7feb6551c640 10 monclient: get_monmap_and_config 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.023+0000 7feb6551c640 10 monclient: build_initial_monmap 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.023+0000 7feb6551c640 10 monclient: monmap: 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:28:17.023975+0000 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:28:17.023975+0000 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:17.023 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:28:17.024 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:28:17.024 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:17.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.023+0000 7feb6551c640 1 Processor -- start 2024-03-25T19:28:17.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.023+0000 7feb6551c640 1 -- start start 2024-03-25T19:28:17.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.023+0000 7feb6551c640 10 monclient: init 2024-03-25T19:28:17.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.024+0000 7feb6551c640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:17.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.024+0000 7feb6551c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:17.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.024+0000 7feb6551c640 10 monclient(hunting): picked mon.noname-a con 0x7feb600a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:17.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.024+0000 7feb6551c640 10 monclient(hunting): picked mon.noname-b con 0x7feb6013f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:17.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.024+0000 7feb6551c640 10 monclient(hunting): picked mon.noname-c con 0x7feb601430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:17.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.024+0000 7feb6551c640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb60142520 con 0x7feb600a98f0 2024-03-25T19:28:17.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.024+0000 7feb6551c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb600b28a0 con 0x7feb601430c0 2024-03-25T19:28:17.026 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.024+0000 7feb6551c640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb600b2a30 con 0x7feb6013f710 2024-03-25T19:28:17.026 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.024+0000 7feb6551c640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:17.026 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.024+0000 7feb6551c640 10 monclient(hunting): authenticate will time out at 5934.384350s 2024-03-25T19:28:17.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.029+0000 7feb5f7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7feb601430c0 0x7feb6014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:55440/0 (socket says 172.21.15.123:55440) 2024-03-25T19:28:17.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.029+0000 7feb5f7fe640 1 -- 172.21.15.123:0/705108661 learned_addr learned my addr 172.21.15.123:0/705108661 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:17.051 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.051+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810495293 0 0) 0x7feb600b28a0 con 0x7feb601430c0 2024-03-25T19:28:17.051 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.051+0000 7feb5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:17.051 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.051+0000 7feb5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:17.051 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.051+0000 7feb5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:17.051 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.051+0000 7feb5dffb640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:28:17.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.051+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb48003520 con 0x7feb601430c0 2024-03-25T19:28:17.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.055+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821788489 0 0) 0x7feb600b2a30 con 0x7feb6013f710 2024-03-25T19:28:17.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.055+0000 7feb5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:17.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.055+0000 7feb5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:17.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.055+0000 7feb5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:17.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.055+0000 7feb5dffb640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:28:17.055 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.055+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb600b28a0 con 0x7feb6013f710 2024-03-25T19:28:17.064 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328642079 0 0) 0x7feb60142520 con 0x7feb600a98f0 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb600b2a30 con 0x7feb600a98f0 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2078746057 0 0) 0x7feb600b28a0 con 0x7feb6013f710 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 >> v1:172.21.15.151:6789/0 conn(0x7feb601430c0 legacy=0x7feb6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 >> v1:172.21.15.123:6789/0 conn(0x7feb600a98f0 legacy=0x7feb600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 1 monclient: found mon.noname-b 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:17.065 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb600b44c0 con 0x7feb6013f710 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7feb54001670 con 0x7feb6013f710 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb6551c640 5 monclient: authenticate success, global_id 4493 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb6551c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb5dffb640 10 monclient: dump: 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:17.066 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:17.067 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:17.067 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:17.067 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:17.067 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:17.067 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.064+0000 7feb6551c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:17.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.074+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb540017d0 con 0x7feb6013f710 2024-03-25T19:28:17.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.074+0000 7feb5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:17.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.074+0000 7feb5dffb640 1 -- 172.21.15.123:0/705108661 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7feb54001be0 con 0x7feb6013f710 2024-03-25T19:28:17.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.074+0000 7feb6551c640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.074+0000 7feb6551c640 4 set_mon_vals no callback set 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.074+0000 7feb5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.074+0000 7feb5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.074+0000 7feb5dffb640 10 monclient: dump: 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:17.075 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:17.077 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.075+0000 7feb6551c640 10 monclient: shutdown 2024-03-25T19:28:17.077 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.075+0000 7feb6551c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:17.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.075+0000 7feb6551c640 1 -- 172.21.15.123:0/705108661 >> v1:172.21.15.123:6790/0 conn(0x7feb6013f710 legacy=0x7feb601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:17.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.075+0000 7feb6551c640 1 -- 172.21.15.123:0/705108661 shutdown_connections 2024-03-25T19:28:17.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.075+0000 7feb6551c640 1 -- 172.21.15.123:0/705108661 >> 172.21.15.123:0/705108661 conn(0x7feb600b7ea0 msgr2=0x7feb600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:17.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.075+0000 7feb6551c640 1 -- 172.21.15.123:0/705108661 shutdown_connections 2024-03-25T19:28:17.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.075+0000 7feb6551c640 1 -- 172.21.15.123:0/705108661 wait complete. 2024-03-25T19:28:17.078 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.076+0000 7feb6551c640 10 monclient: build_initial_monmap 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.076+0000 7feb6551c640 10 monclient: monmap: 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.076+0000 7feb6551c640 1 librados: starting msgr at 2024-03-25T19:28:17.079 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.076+0000 7feb6551c640 1 librados: starting objecter 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.076+0000 7feb6551c640 1 Processor -- start 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.076+0000 7feb6551c640 1 -- start start 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.076+0000 7feb6551c640 1 librados: setting wanted keys 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.076+0000 7feb6551c640 1 librados: calling monclient init 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.076+0000 7feb6551c640 10 monclient: init 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.077+0000 7feb6551c640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.077+0000 7feb6551c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.077+0000 7feb6551c640 10 monclient(hunting): picked mon.noname-b con 0x7feb600a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.077+0000 7feb6551c640 10 monclient(hunting): picked mon.noname-a con 0x7feb6013f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.077+0000 7feb6551c640 10 monclient(hunting): picked mon.noname-c con 0x7feb601430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.077+0000 7feb6551c640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb60136fa0 con 0x7feb6013f710 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.077+0000 7feb6551c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6013b780 con 0x7feb600a98f0 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.077+0000 7feb6551c640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb601d7ba0 con 0x7feb601430c0 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.077+0000 7feb6551c640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:17.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.077+0000 7feb6551c640 10 monclient(hunting): authenticate will time out at 5934.437337s 2024-03-25T19:28:17.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.093+0000 7feb5f7fe640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7feb601430c0 0x7feb601e1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:40072/0 (socket says 172.21.15.123:40072) 2024-03-25T19:28:17.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.093+0000 7feb5f7fe640 1 -- 172.21.15.123:0/380030648 learned_addr learned my addr 172.21.15.123:0/380030648 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:17.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.095+0000 7feb477fe640 10 client.?.objecter ms_handle_connect 0x7feb601430c0 2024-03-25T19:28:17.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.095+0000 7feb477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:17.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.099+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296104433 0 0) 0x7feb601d7ba0 con 0x7feb601430c0 2024-03-25T19:28:17.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.099+0000 7feb477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:17.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.099+0000 7feb477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:17.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.099+0000 7feb477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:17.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.099+0000 7feb477fe640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:28:17.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.099+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb4c004eb0 con 0x7feb601430c0 2024-03-25T19:28:17.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.104+0000 7feb477fe640 10 client.?.objecter ms_handle_connect 0x7feb6013f710 2024-03-25T19:28:17.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.104+0000 7feb477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:17.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.105+0000 7feb477fe640 10 client.?.objecter ms_handle_connect 0x7feb600a98f0 2024-03-25T19:28:17.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.105+0000 7feb477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:17.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.115+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 392516364 0 0) 0x7feb4c004eb0 con 0x7feb601430c0 2024-03-25T19:28:17.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.115+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb601d7ba0 con 0x7feb601430c0 2024-03-25T19:28:17.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.115+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7feb58001b30 con 0x7feb601430c0 2024-03-25T19:28:17.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.115+0000 7feb477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:17.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.115+0000 7feb477fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.115+0000 7feb477fe640 10 monclient(hunting): dump: 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:17.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.115+0000 7feb477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:17.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.118+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543760321 0 0) 0x7feb60136fa0 con 0x7feb6013f710 2024-03-25T19:28:17.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.118+0000 7feb477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:17.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.118+0000 7feb477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:17.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.118+0000 7feb477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:17.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.118+0000 7feb477fe640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:28:17.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.118+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb4c004eb0 con 0x7feb6013f710 2024-03-25T19:28:17.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.120+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662653747 0 0) 0x7feb6013b780 con 0x7feb600a98f0 2024-03-25T19:28:17.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.120+0000 7feb477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:17.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.120+0000 7feb477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:17.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.120+0000 7feb477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:17.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.120+0000 7feb477fe640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:28:17.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.120+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb60136fa0 con 0x7feb600a98f0 2024-03-25T19:28:17.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 227587583 0 0) 0x7feb601d7ba0 con 0x7feb601430c0 2024-03-25T19:28:17.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 >> v1:172.21.15.151:6789/0 conn(0x7feb600a98f0 legacy=0x7feb60136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:17.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 >> v1:172.21.15.123:6789/0 conn(0x7feb6013f710 legacy=0x7feb601d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:17.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:17.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 1 monclient: found mon.c 2024-03-25T19:28:17.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:17.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:17.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb60137990 con 0x7feb601430c0 2024-03-25T19:28:17.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:17.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:17.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:17.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb6551c640 5 monclient: authenticate success, global_id 4496 2024-03-25T19:28:17.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb6551c640 10 monclient: _renew_subs 2024-03-25T19:28:17.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb6551c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:17.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb6551c640 1 -- 172.21.15.123:0/380030648 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb60137c40 con 0x7feb601430c0 2024-03-25T19:28:17.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb6551c640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:17.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb6551c640 10 monclient: _renew_subs 2024-03-25T19:28:17.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb6551c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:17.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb6551c640 1 -- 172.21.15.123:0/380030648 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb60139860 con 0x7feb601430c0 2024-03-25T19:28:17.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.122+0000 7feb6551c640 1 librados: init done 2024-03-25T19:28:17.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.123+0000 7feb6551c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:17.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.123+0000 7feb6551c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:17.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.123+0000 7feb6551c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:17.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.123+0000 7feb6551c640 1 -- 172.21.15.123:0/380030648 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb601375f0 con 0x7feb601430c0 2024-03-25T19:28:17.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.427+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb58002020 con 0x7feb601430c0 2024-03-25T19:28:17.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.427+0000 7feb477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:17.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.427+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7feb58004d00 con 0x7feb601430c0 2024-03-25T19:28:17.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.427+0000 7feb477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:17.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.427+0000 7feb477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:17.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.427+0000 7feb477fe640 10 monclient: dump: 2024-03-25T19:28:17.427 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:17.428 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:17.428 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:17.428 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:17.428 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:17.428 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:17.428 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:17.428 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:17.428 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:17.428 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:17.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.427+0000 7feb5d7fa640 4 set_mon_vals no callback set 2024-03-25T19:28:17.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.428+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7feb58004f20 con 0x7feb601430c0 2024-03-25T19:28:17.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.428+0000 7feb5effd640 10 monclient: get_auth_request con 0x7feb4c06de80 auth_method 0 2024-03-25T19:28:17.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.429+0000 7feb477fe640 10 client.4496.objecter ms_handle_connect 0x7feb4c06de80 2024-03-25T19:28:17.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.433+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7feb58054ad0 con 0x7feb601430c0 2024-03-25T19:28:17.433 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.433+0000 7feb477fe640 10 client.4496.objecter ms_dispatch 0x7feb60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:17.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.433+0000 7feb477fe640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:17.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.433+0000 7feb477fe640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:17.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.433+0000 7feb477fe640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-25T19:28:17.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.433+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb58050410 con 0x7feb601430c0 2024-03-25T19:28:17.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.433+0000 7feb477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:17.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.433+0000 7feb477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:17.678 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.678+0000 7feb6551c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:17.679 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.678+0000 7feb6551c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:17.679 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.678+0000 7feb6551c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:17.679 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.678+0000 7feb6551c640 1 -- 172.21.15.123:0/380030648 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb600627b0 con 0x7feb601430c0 2024-03-25T19:28:17.700 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.700+0000 7feb477fe640 1 -- 172.21.15.123:0/380030648 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7feb58052510 con 0x7feb601430c0 2024-03-25T19:28:17.700 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.700+0000 7feb477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:17.700 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.700+0000 7feb477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:17.700 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:28:17.700 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.702+0000 7feb6551c640 10 librados: watch_flush enter 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.702+0000 7feb6551c640 10 librados: watch_flush exit 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.702+0000 7feb6551c640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.702+0000 7feb6551c640 10 client.4496.objecter successfully canceled tick 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.702+0000 7feb6551c640 1 -- 172.21.15.123:0/380030648 >> v1:172.21.15.151:6800/727684239 conn(0x7feb4c06de80 legacy=0x7feb4c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.703+0000 7feb6551c640 10 monclient: shutdown 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.703+0000 7feb6551c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.703+0000 7feb6551c640 1 -- 172.21.15.123:0/380030648 >> v1:172.21.15.123:6790/0 conn(0x7feb601430c0 legacy=0x7feb601e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.703+0000 7feb6551c640 1 -- 172.21.15.123:0/380030648 shutdown_connections 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.703+0000 7feb6551c640 1 -- 172.21.15.123:0/380030648 >> 172.21.15.123:0/380030648 conn(0x7feb600b7ea0 msgr2=0x7feb6014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:17.703 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.703+0000 7feb6551c640 1 -- 172.21.15.123:0/380030648 shutdown_connections 2024-03-25T19:28:17.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.703+0000 7feb6551c640 1 -- 172.21.15.123:0/380030648 wait complete. 2024-03-25T19:28:17.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:17.703+0000 7feb6551c640 1 librados: shutdown 2024-03-25T19:28:17.715 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:24.521 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:24.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.637+0000 7f83515e8640 10 monclient: get_monmap_and_config 2024-03-25T19:28:24.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.637+0000 7f83515e8640 10 monclient: build_initial_monmap 2024-03-25T19:28:24.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.637+0000 7f83515e8640 10 monclient: monmap: 2024-03-25T19:28:24.637 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:24.638 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:24.638 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:28:24.638349+0000 2024-03-25T19:28:24.638 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:28:24.638349+0000 2024-03-25T19:28:24.638 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:24.638 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:24.638 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:24.638 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:28:24.638 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:28:24.638 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:24.639 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.638+0000 7f83515e8640 1 Processor -- start 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.638+0000 7f83515e8640 1 -- start start 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.638+0000 7f83515e8640 10 monclient: init 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.638+0000 7f83515e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.638+0000 7f83515e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.638+0000 7f83515e8640 10 monclient(hunting): picked mon.noname-c con 0x7f834c0a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.638+0000 7f83515e8640 10 monclient(hunting): picked mon.noname-b con 0x7f834c13f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.638+0000 7f83515e8640 10 monclient(hunting): picked mon.noname-a con 0x7f834c1430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.638+0000 7f83515e8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834c142520 con 0x7f834c1430c0 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.638+0000 7f83515e8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834c0b28a0 con 0x7f834c0a98f0 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.639+0000 7f83515e8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834c0b2a30 con 0x7f834c13f710 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.639+0000 7f83515e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:24.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.639+0000 7f83515e8640 10 monclient(hunting): authenticate will time out at 5941.998943s 2024-03-25T19:28:24.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.644+0000 7f834affd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f834c0a98f0 0x7f834c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:60226/0 (socket says 172.21.15.123:60226) 2024-03-25T19:28:24.644 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.645+0000 7f834affd640 1 -- 172.21.15.123:0/1920078486 learned_addr learned my addr 172.21.15.123:0/1920078486 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:24.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.652+0000 7f8349ffb640 1 -- 172.21.15.123:0/1920078486 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131147639 0 0) 0x7f834c0b28a0 con 0x7f834c0a98f0 2024-03-25T19:28:24.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.652+0000 7f8349ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.652+0000 7f8349ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.652 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.652+0000 7f8349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.653 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.652+0000 7f8349ffb640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:28:24.653 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.652+0000 7f8349ffb640 1 -- 172.21.15.123:0/1920078486 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8334003520 con 0x7f834c0a98f0 2024-03-25T19:28:24.660 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 1 -- 172.21.15.123:0/1920078486 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1926842050 0 0) 0x7f8334003520 con 0x7f834c0a98f0 2024-03-25T19:28:24.660 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 1 -- 172.21.15.123:0/1920078486 >> v1:172.21.15.123:6790/0 conn(0x7f834c13f710 legacy=0x7f834c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.660 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 1 -- 172.21.15.123:0/1920078486 >> v1:172.21.15.123:6789/0 conn(0x7f834c1430c0 legacy=0x7f834c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 1 monclient: found mon.noname-c 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 1 -- 172.21.15.123:0/1920078486 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c0b44c0 con 0x7f834c0a98f0 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 1 -- 172.21.15.123:0/1920078486 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f833c001690 con 0x7f834c0a98f0 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f8349ffb640 10 monclient: dump: 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:24.661 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:24.662 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.662 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:24.662 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:24.662 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:24.662 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:24.662 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:24.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f83515e8640 5 monclient: authenticate success, global_id 4477 2024-03-25T19:28:24.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.660+0000 7f83515e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:24.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.664+0000 7f8349ffb640 1 -- 172.21.15.123:0/1920078486 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f833c001860 con 0x7f834c0a98f0 2024-03-25T19:28:24.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.664+0000 7f8349ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:24.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.664+0000 7f83515e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:24.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.664+0000 7f83515e8640 4 set_mon_vals no callback set 2024-03-25T19:28:24.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.665+0000 7f83515e8640 10 monclient: shutdown 2024-03-25T19:28:24.666 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.665+0000 7f83515e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.665+0000 7f83515e8640 1 -- 172.21.15.123:0/1920078486 >> v1:172.21.15.151:6789/0 conn(0x7f834c0a98f0 legacy=0x7f834c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.665+0000 7f83515e8640 1 -- 172.21.15.123:0/1920078486 shutdown_connections 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.665+0000 7f83515e8640 1 -- 172.21.15.123:0/1920078486 >> 172.21.15.123:0/1920078486 conn(0x7f834c0b7ea0 msgr2=0x7f834c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.665+0000 7f83515e8640 1 -- 172.21.15.123:0/1920078486 shutdown_connections 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.665+0000 7f83515e8640 1 -- 172.21.15.123:0/1920078486 wait complete. 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.665+0000 7f83515e8640 10 monclient: build_initial_monmap 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.665+0000 7f83515e8640 10 monclient: monmap: 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:28:24.667 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:24.668 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:28:24.668 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:28:24.668 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:24.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.666+0000 7f83515e8640 1 librados: starting msgr at 2024-03-25T19:28:24.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.666+0000 7f83515e8640 1 librados: starting objecter 2024-03-25T19:28:24.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.666+0000 7f83515e8640 1 Processor -- start 2024-03-25T19:28:24.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.666+0000 7f83515e8640 1 -- start start 2024-03-25T19:28:24.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.666+0000 7f83515e8640 1 librados: setting wanted keys 2024-03-25T19:28:24.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.666+0000 7f83515e8640 1 librados: calling monclient init 2024-03-25T19:28:24.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.666+0000 7f83515e8640 10 monclient: init 2024-03-25T19:28:24.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.666+0000 7f83515e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:24.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.667+0000 7f83515e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:24.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.667+0000 7f83515e8640 10 monclient(hunting): picked mon.noname-c con 0x7f834c0a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:24.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.667+0000 7f83515e8640 10 monclient(hunting): picked mon.noname-b con 0x7f834c13f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:24.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.667+0000 7f83515e8640 10 monclient(hunting): picked mon.noname-a con 0x7f834c1430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:24.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.667+0000 7f83515e8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834c12f5f0 con 0x7f834c1430c0 2024-03-25T19:28:24.669 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.667+0000 7f83515e8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834c1db990 con 0x7f834c13f710 2024-03-25T19:28:24.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.667+0000 7f83515e8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834c0b71c0 con 0x7f834c0a98f0 2024-03-25T19:28:24.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.667+0000 7f83515e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:24.670 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.667+0000 7f83515e8640 10 monclient(hunting): authenticate will time out at 5942.026973s 2024-03-25T19:28:24.678 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.678+0000 7f834b7fe640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f834c1430c0 0x7f834c1db280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:42492/0 (socket says 172.21.15.123:42492) 2024-03-25T19:28:24.678 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.678+0000 7f834b7fe640 1 -- 172.21.15.123:0/3312645906 learned_addr learned my addr 172.21.15.123:0/3312645906 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:24.679 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.680+0000 7f83337fe640 10 client.?.objecter ms_handle_connect 0x7f834c1430c0 2024-03-25T19:28:24.679 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.680+0000 7f83337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.685+0000 7f83337fe640 10 client.?.objecter ms_handle_connect 0x7f834c0a98f0 2024-03-25T19:28:24.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.685+0000 7f83337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.685+0000 7f83337fe640 10 client.?.objecter ms_handle_connect 0x7f834c13f710 2024-03-25T19:28:24.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.685+0000 7f83337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.701 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.701+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066309292 0 0) 0x7f834c12f5f0 con 0x7f834c1430c0 2024-03-25T19:28:24.701 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.701+0000 7f83337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.701 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.701+0000 7f83337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.701 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.701+0000 7f83337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.701 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.701+0000 7f83337fe640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:28:24.701 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.701+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8338004eb0 con 0x7f834c1430c0 2024-03-25T19:28:24.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.705+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641789959 0 0) 0x7f834c0b71c0 con 0x7f834c0a98f0 2024-03-25T19:28:24.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.705+0000 7f83337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.705+0000 7f83337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.705+0000 7f83337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.705+0000 7f83337fe640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:28:24.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.705+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f834c12f5f0 con 0x7f834c0a98f0 2024-03-25T19:28:24.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.708+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947539909 0 0) 0x7f834c1db990 con 0x7f834c13f710 2024-03-25T19:28:24.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.708+0000 7f83337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.708+0000 7f83337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.708+0000 7f83337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.708+0000 7f83337fe640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:28:24.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.708+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f834c0b71c0 con 0x7f834c13f710 2024-03-25T19:28:24.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.717+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4125525933 0 0) 0x7f8338004eb0 con 0x7f834c1430c0 2024-03-25T19:28:24.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.717+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f834c1db990 con 0x7f834c1430c0 2024-03-25T19:28:24.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.717+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8344001ad0 con 0x7f834c1430c0 2024-03-25T19:28:24.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.717+0000 7f83337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.717+0000 7f83337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:24.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.717+0000 7f83337fe640 10 monclient(hunting): dump: 2024-03-25T19:28:24.717 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:24.718 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:24.718 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:24.718 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:24.718 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.718 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:24.718 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:24.718 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:24.718 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:24.718 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:24.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.717+0000 7f83337fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:24.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.730+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1019386771 0 0) 0x7f834c12f5f0 con 0x7f834c0a98f0 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.731+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8338004eb0 con 0x7f834c0a98f0 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.731+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f833c001bb0 con 0x7f834c0a98f0 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.731+0000 7f83337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.731+0000 7f83337fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.731+0000 7f83337fe640 10 monclient(hunting): dump: 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:24.731 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:24.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.733+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2044538731 0 0) 0x7f834c0b71c0 con 0x7f834c13f710 2024-03-25T19:28:24.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.733+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f834c12f5f0 con 0x7f834c13f710 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.733+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8340001830 con 0x7f834c13f710 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.734+0000 7f83337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.734+0000 7f83337fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.734+0000 7f83337fe640 10 monclient(hunting): dump: 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:24.734 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:24.735 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3146605795 0 0) 0x7f8338004eb0 con 0x7f834c0a98f0 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 >> v1:172.21.15.151:6789/0 conn(0x7f834c13f710 legacy=0x7f834c1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 >> v1:172.21.15.123:6789/0 conn(0x7f834c1430c0 legacy=0x7f834c1db280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 1 monclient: found mon.c 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c13b180 con 0x7f834c0a98f0 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:24.740 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:24.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83515e8640 5 monclient: authenticate success, global_id 4502 2024-03-25T19:28:24.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83515e8640 10 monclient: _renew_subs 2024-03-25T19:28:24.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83515e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:24.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.739+0000 7f83515e8640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834c13b460 con 0x7f834c0a98f0 2024-03-25T19:28:24.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.740+0000 7f83515e8640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:24.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.740+0000 7f83515e8640 10 monclient: _renew_subs 2024-03-25T19:28:24.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.740+0000 7f83515e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:24.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.740+0000 7f83515e8640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f834c139790 con 0x7f834c0a98f0 2024-03-25T19:28:24.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.740+0000 7f83515e8640 1 librados: init done 2024-03-25T19:28:24.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.741+0000 7f83515e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.741+0000 7f83515e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.741+0000 7f83515e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:24.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:24.741+0000 7f83515e8640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834c139a30 con 0x7f834c0a98f0 2024-03-25T19:28:25.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.035+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f833c004a90 con 0x7f834c0a98f0 2024-03-25T19:28:25.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.035+0000 7f83337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:25.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.035+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f833c001250 con 0x7f834c0a98f0 2024-03-25T19:28:25.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.035+0000 7f83337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.035+0000 7f83337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:25.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.035+0000 7f83337fe640 10 monclient: dump: 2024-03-25T19:28:25.035 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:25.036 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:25.036 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:25.036 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:25.036 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.036 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:25.036 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:25.036 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:25.036 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:25.036 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:25.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.035+0000 7f83497fa640 4 set_mon_vals no callback set 2024-03-25T19:28:25.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.036+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f833c01ac10 con 0x7f834c0a98f0 2024-03-25T19:28:25.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.036+0000 7f834a7fc640 10 monclient: get_auth_request con 0x7f833806e2b0 auth_method 0 2024-03-25T19:28:25.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.037+0000 7f83337fe640 10 client.4502.objecter ms_handle_connect 0x7f833806e2b0 2024-03-25T19:28:25.041 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.041+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f833c0547c0 con 0x7f834c0a98f0 2024-03-25T19:28:25.041 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.041+0000 7f83337fe640 10 client.4502.objecter ms_dispatch 0x7f834c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:25.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.041+0000 7f83337fe640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:25.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.041+0000 7f83337fe640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:25.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.041+0000 7f83337fe640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-25T19:28:25.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.041+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f833c0502d0 con 0x7f834c0a98f0 2024-03-25T19:28:25.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.041+0000 7f83337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:25.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.041+0000 7f83337fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:25.263 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.263+0000 7f83515e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:25.263 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.263+0000 7f83515e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:25.263 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.263+0000 7f83515e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:25.263 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.263+0000 7f83515e8640 1 -- 172.21.15.123:0/3312645906 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f834c14f2d0 con 0x7f834c0a98f0 2024-03-25T19:28:25.314 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.314+0000 7f83337fe640 1 -- 172.21.15.123:0/3312645906 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f833c052200 con 0x7f834c0a98f0 2024-03-25T19:28:25.314 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.314+0000 7f83337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:25.314 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.314+0000 7f83337fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:25.314 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:28:25.314 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 10 librados: watch_flush enter 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 10 librados: watch_flush exit 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 10 client.4502.objecter successfully canceled tick 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 1 -- 172.21.15.123:0/3312645906 >> v1:172.21.15.151:6800/727684239 conn(0x7f833806e2b0 legacy=0x7f8338070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 10 monclient: shutdown 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 1 -- 172.21.15.123:0/3312645906 >> v1:172.21.15.123:6790/0 conn(0x7f834c0a98f0 legacy=0x7f834c12eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 1 -- 172.21.15.123:0/3312645906 shutdown_connections 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 1 -- 172.21.15.123:0/3312645906 >> 172.21.15.123:0/3312645906 conn(0x7f834c0b7ea0 msgr2=0x7f834c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 1 -- 172.21.15.123:0/3312645906 shutdown_connections 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 1 -- 172.21.15.123:0/3312645906 wait complete. 2024-03-25T19:28:25.317 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:25.316+0000 7f83515e8640 1 librados: shutdown 2024-03-25T19:28:25.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:30.723 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:06:20.295 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:30.768 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:06:20.301 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:30.769 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:06:20.342 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:30.771 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:06:20.304 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:30.802 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:06:20.375 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:30.817 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:06:20.350 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:30.830 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:06:20.402 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:30.845 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:06:20.417 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:30.869 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:06:20.441 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:32.135 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:32.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.246+0000 7fa6bc445640 10 monclient: get_monmap_and_config 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.246+0000 7fa6bc445640 10 monclient: build_initial_monmap 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.246+0000 7fa6bc445640 10 monclient: monmap: 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:28:32.247836+0000 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:28:32.247836+0000 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:28:32.247 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:32.248 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.247+0000 7fa6bc445640 1 Processor -- start 2024-03-25T19:28:32.248 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.247+0000 7fa6bc445640 1 -- start start 2024-03-25T19:28:32.248 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.247+0000 7fa6bc445640 10 monclient: init 2024-03-25T19:28:32.248 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.248+0000 7fa6bc445640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:32.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.248+0000 7fa6bc445640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:32.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.248+0000 7fa6bc445640 10 monclient(hunting): picked mon.noname-b con 0x7fa6b40a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:32.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.248+0000 7fa6bc445640 10 monclient(hunting): picked mon.noname-a con 0x7fa6b413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:32.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.248+0000 7fa6bc445640 10 monclient(hunting): picked mon.noname-c con 0x7fa6b41430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:32.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.248+0000 7fa6bc445640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b4142520 con 0x7fa6b413f710 2024-03-25T19:28:32.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.248+0000 7fa6bc445640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b40b28a0 con 0x7fa6b41430c0 2024-03-25T19:28:32.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.248+0000 7fa6bc445640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b40b2a30 con 0x7fa6b40a98f0 2024-03-25T19:28:32.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.248+0000 7fa6bc445640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:32.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.248+0000 7fa6bc445640 10 monclient(hunting): authenticate will time out at 5949.608845s 2024-03-25T19:28:32.255 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.255+0000 7fa6b99e0640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa6b413f710 0x7fa6b41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:57380/0 (socket says 172.21.15.123:57380) 2024-03-25T19:28:32.255 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.255+0000 7fa6b99e0640 1 -- 172.21.15.123:0/2912712507 learned_addr learned my addr 172.21.15.123:0/2912712507 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:32.270 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.270+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136016164 0 0) 0x7fa6b4142520 con 0x7fa6b413f710 2024-03-25T19:28:32.271 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.271+0000 7fa6b91df640 10 monclient(hunting): _negotiate 2024-03-25T19:28:32.271 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.271+0000 7fa6b91df640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:32.271 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.271+0000 7fa6b91df640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:32.271 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.271+0000 7fa6b91df640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:28:32.271 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.271+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a0003520 con 0x7fa6b413f710 2024-03-25T19:28:32.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.272+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550254700 0 0) 0x7fa6b40b28a0 con 0x7fa6b41430c0 2024-03-25T19:28:32.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.272+0000 7fa6b91df640 10 monclient(hunting): _negotiate 2024-03-25T19:28:32.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.272+0000 7fa6b91df640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:32.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.272+0000 7fa6b91df640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:32.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.272+0000 7fa6b91df640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:28:32.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.272+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b4142520 con 0x7fa6b41430c0 2024-03-25T19:28:32.277 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3191777918 0 0) 0x7fa6a0003520 con 0x7fa6b413f710 2024-03-25T19:28:32.277 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 >> v1:172.21.15.123:6790/0 conn(0x7fa6b40a98f0 legacy=0x7fa6b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.277 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 >> v1:172.21.15.151:6789/0 conn(0x7fa6b41430c0 legacy=0x7fa6b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 10 monclient: _finish_hunting 0 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 1 monclient: found mon.noname-a 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b40b44c0 con 0x7fa6b413f710 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 10 monclient: _finish_auth 0 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 10 monclient: _check_auth_tickets 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6bc445640 5 monclient: authenticate success, global_id 4473 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6bc445640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa6a4001690 con 0x7fa6b413f710 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:32.278 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6b91df640 10 monclient: dump: 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:32.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.277+0000 7fa6bc445640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.285+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6a40017f0 con 0x7fa6b413f710 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.285+0000 7fa6b91df640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.285+0000 7fa6b91df640 1 -- 172.21.15.123:0/2912712507 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa6a4001c00 con 0x7fa6b413f710 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.285+0000 7fa6b91df640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.285+0000 7fa6b91df640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.285+0000 7fa6b91df640 10 monclient: dump: 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:32.285 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:32.286 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:32.286 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:32.286 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:32.286 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:32.286 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:32.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.285+0000 7fa6bc445640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:32.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.285+0000 7fa6bc445640 4 set_mon_vals no callback set 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.286+0000 7fa6bc445640 10 monclient: shutdown 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.286+0000 7fa6bc445640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.286+0000 7fa6bc445640 1 -- 172.21.15.123:0/2912712507 >> v1:172.21.15.123:6789/0 conn(0x7fa6b413f710 legacy=0x7fa6b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.286+0000 7fa6bc445640 1 -- 172.21.15.123:0/2912712507 shutdown_connections 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.286+0000 7fa6bc445640 1 -- 172.21.15.123:0/2912712507 >> 172.21.15.123:0/2912712507 conn(0x7fa6b40b7ea0 msgr2=0x7fa6b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.286+0000 7fa6bc445640 1 -- 172.21.15.123:0/2912712507 shutdown_connections 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.286+0000 7fa6bc445640 1 -- 172.21.15.123:0/2912712507 wait complete. 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.286+0000 7fa6bc445640 10 monclient: build_initial_monmap 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.286+0000 7fa6bc445640 10 monclient: monmap: 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:28:32.288 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 1 librados: starting msgr at 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 1 librados: starting objecter 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 1 Processor -- start 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 1 -- start start 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 1 librados: setting wanted keys 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 1 librados: calling monclient init 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 10 monclient: init 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:32.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:32.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 10 monclient(hunting): picked mon.noname-b con 0x7fa6b40a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:32.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 10 monclient(hunting): picked mon.noname-a con 0x7fa6b413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:32.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 10 monclient(hunting): picked mon.noname-c con 0x7fa6b41430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:32.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b40abd70 con 0x7fa6b413f710 2024-03-25T19:28:32.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b41d7710 con 0x7fa6b40a98f0 2024-03-25T19:28:32.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b41d8340 con 0x7fa6b41430c0 2024-03-25T19:28:32.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:32.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.287+0000 7fa6bc445640 10 monclient(hunting): authenticate will time out at 5949.647741s 2024-03-25T19:28:32.303 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.304+0000 7fa6ba9e2640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fa6b41430c0 0x7fa6b41d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:44522/0 (socket says 172.21.15.123:44522) 2024-03-25T19:28:32.303 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.304+0000 7fa6ba9e2640 1 -- 172.21.15.123:0/99690504 learned_addr learned my addr 172.21.15.123:0/99690504 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:32.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.306+0000 7fa6aaffd640 10 client.?.objecter ms_handle_connect 0x7fa6b41430c0 2024-03-25T19:28:32.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.306+0000 7fa6aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:32.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.309+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121435294 0 0) 0x7fa6b41d8340 con 0x7fa6b41430c0 2024-03-25T19:28:32.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.309+0000 7fa6aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:32.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.309+0000 7fa6aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:32.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.309+0000 7fa6aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:32.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.309+0000 7fa6aaffd640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:28:32.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.309+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa698004eb0 con 0x7fa6b41430c0 2024-03-25T19:28:32.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.315+0000 7fa6aaffd640 10 client.?.objecter ms_handle_connect 0x7fa6b40a98f0 2024-03-25T19:28:32.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.315+0000 7fa6aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:32.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.318+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1202543843 0 0) 0x7fa698004eb0 con 0x7fa6b41430c0 2024-03-25T19:28:32.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.318+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6b41d8340 con 0x7fa6b41430c0 2024-03-25T19:28:32.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.318+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa6ac001c90 con 0x7fa6b41430c0 2024-03-25T19:28:32.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.318+0000 7fa6aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:32.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.318+0000 7fa6aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:32.318 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.318+0000 7fa6aaffd640 10 monclient(hunting): dump: 2024-03-25T19:28:32.318 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:32.319 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:32.319 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:32.319 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:32.319 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:32.319 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:32.319 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:32.319 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:32.319 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:32.319 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:32.319 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.318+0000 7fa6aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:32.325 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.325+0000 7fa6aaffd640 10 client.?.objecter ms_handle_connect 0x7fa6b413f710 2024-03-25T19:28:32.325 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.325+0000 7fa6aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:32.326 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.326+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048377408 0 0) 0x7fa6b41d7710 con 0x7fa6b40a98f0 2024-03-25T19:28:32.326 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.326+0000 7fa6aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:32.326 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.326+0000 7fa6aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:32.326 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.326+0000 7fa6aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:32.326 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.326+0000 7fa6aaffd640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:28:32.326 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.326+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa698004eb0 con 0x7fa6b40a98f0 2024-03-25T19:28:32.334 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1549305307 0 0) 0x7fa6b41d8340 con 0x7fa6b41430c0 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 >> v1:172.21.15.151:6789/0 conn(0x7fa6b40a98f0 legacy=0x7fa6b4137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 >> v1:172.21.15.123:6789/0 conn(0x7fa6b413f710 legacy=0x7fa6b41d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 1 monclient: found mon.c 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b41d84d0 con 0x7fa6b41430c0 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6bc445640 5 monclient: authenticate success, global_id 4508 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6bc445640 10 monclient: _renew_subs 2024-03-25T19:28:32.335 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6bc445640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:32.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6bc445640 1 -- 172.21.15.123:0/99690504 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b41d41e0 con 0x7fa6b41430c0 2024-03-25T19:28:32.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6bc445640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:32.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6bc445640 10 monclient: _renew_subs 2024-03-25T19:28:32.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6bc445640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:32.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6bc445640 1 -- 172.21.15.123:0/99690504 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b41d5150 con 0x7fa6b41430c0 2024-03-25T19:28:32.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.334+0000 7fa6bc445640 1 librados: init done 2024-03-25T19:28:32.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.335+0000 7fa6bc445640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:32.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.335+0000 7fa6bc445640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:32.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.335+0000 7fa6bc445640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:32.336 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.335+0000 7fa6bc445640 1 -- 172.21.15.123:0/99690504 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b40627b0 con 0x7fa6b41430c0 2024-03-25T19:28:32.548 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.548+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6ac003990 con 0x7fa6b41430c0 2024-03-25T19:28:32.548 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.548+0000 7fa6aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:32.548 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.548+0000 7fa6b89de640 4 set_mon_vals no callback set 2024-03-25T19:28:32.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.632+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa6ac003d60 con 0x7fa6b41430c0 2024-03-25T19:28:32.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.632+0000 7fa6aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:32.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.632+0000 7fa6aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:32.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.632+0000 7fa6aaffd640 10 monclient: dump: 2024-03-25T19:28:32.632 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:32.632 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:32.633 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:32.633 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:32.633 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:32.633 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:32.633 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:32.633 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:32.633 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:32.633 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:32.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.633+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fa6ac003f40 con 0x7fa6b41430c0 2024-03-25T19:28:32.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.633+0000 7fa6ba1e1640 10 monclient: get_auth_request con 0x7fa69806dd60 auth_method 0 2024-03-25T19:28:32.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.634+0000 7fa6aaffd640 10 client.4508.objecter ms_handle_connect 0x7fa69806dd60 2024-03-25T19:28:32.639 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.639+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fa6ac054bc0 con 0x7fa6b41430c0 2024-03-25T19:28:32.639 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.639+0000 7fa6aaffd640 10 client.4508.objecter ms_dispatch 0x7fa6b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:32.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.639+0000 7fa6aaffd640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:32.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.639+0000 7fa6aaffd640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:32.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.639+0000 7fa6aaffd640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-25T19:28:32.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.639+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6ac0506d0 con 0x7fa6b41430c0 2024-03-25T19:28:32.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.639+0000 7fa6aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:32.640 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.639+0000 7fa6aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:32.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.896+0000 7fa6bc445640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:32.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.896+0000 7fa6bc445640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:32.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.896+0000 7fa6bc445640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:28:32.897 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:32.896+0000 7fa6bc445640 1 -- 172.21.15.123:0/99690504 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6b41d78a0 con 0x7fa6b41430c0 2024-03-25T19:28:33.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.009+0000 7fa6aaffd640 1 -- 172.21.15.123:0/99690504 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fa6ac052600 con 0x7fa6b41430c0 2024-03-25T19:28:33.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.009+0000 7fa6aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:33.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.009+0000 7fa6aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:33.009 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:28:33.009 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:33.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 10 librados: watch_flush enter 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 10 librados: watch_flush exit 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 10 client.4508.objecter successfully canceled tick 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 1 -- 172.21.15.123:0/99690504 >> v1:172.21.15.151:6800/727684239 conn(0x7fa69806dd60 legacy=0x7fa698070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 10 monclient: shutdown 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 1 -- 172.21.15.123:0/99690504 >> v1:172.21.15.123:6790/0 conn(0x7fa6b41430c0 legacy=0x7fa6b41d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 1 -- 172.21.15.123:0/99690504 shutdown_connections 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 1 -- 172.21.15.123:0/99690504 >> 172.21.15.123:0/99690504 conn(0x7fa6b40b7ea0 msgr2=0x7fa6b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 1 -- 172.21.15.123:0/99690504 shutdown_connections 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 1 -- 172.21.15.123:0/99690504 wait complete. 2024-03-25T19:28:33.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:33.011+0000 7fa6bc445640 1 librados: shutdown 2024-03-25T19:28:33.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:39.830 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:39.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.932+0000 7ffbe8d3d640 10 monclient: get_monmap_and_config 2024-03-25T19:28:39.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.932+0000 7ffbe8d3d640 10 monclient: build_initial_monmap 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.932+0000 7ffbe8d3d640 10 monclient: monmap: 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:28:39.933294+0000 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:28:39.933294+0000 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:39.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.932+0000 7ffbe8d3d640 1 Processor -- start 2024-03-25T19:28:39.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 1 -- start start 2024-03-25T19:28:39.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 10 monclient: init 2024-03-25T19:28:39.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:39.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:39.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 10 monclient(hunting): picked mon.noname-a con 0x7ffbe40b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:39.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 10 monclient(hunting): picked mon.noname-b con 0x7ffbe40b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:39.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 10 monclient(hunting): picked mon.noname-c con 0x7ffbe414be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:39.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe40af940 con 0x7ffbe40b4d60 2024-03-25T19:28:39.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe40b0a70 con 0x7ffbe414be90 2024-03-25T19:28:39.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe40b0d50 con 0x7ffbe40b1730 2024-03-25T19:28:39.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:39.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.933+0000 7ffbe8d3d640 10 monclient(hunting): authenticate will time out at 5957.293678s 2024-03-25T19:28:39.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.938+0000 7ffbe2d9d640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ffbe414be90 0x7ffbe414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:57310/0 (socket says 172.21.15.123:57310) 2024-03-25T19:28:39.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.938+0000 7ffbe2d9d640 1 -- 172.21.15.123:0/26337477 learned_addr learned my addr 172.21.15.123:0/26337477 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:39.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.950+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264572585 0 0) 0x7ffbe40b0a70 con 0x7ffbe414be90 2024-03-25T19:28:39.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.950+0000 7ffbe159a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.950+0000 7ffbe159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.950+0000 7ffbe159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.950+0000 7ffbe159a640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:28:39.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.950+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc003520 con 0x7ffbe414be90 2024-03-25T19:28:39.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.959+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844324874 0 0) 0x7ffbe40b0d50 con 0x7ffbe40b1730 2024-03-25T19:28:39.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.959+0000 7ffbe159a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.959+0000 7ffbe159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.959+0000 7ffbe159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.959+0000 7ffbe159a640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:28:39.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.959+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbe40b0a70 con 0x7ffbe40b1730 2024-03-25T19:28:39.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4073141347 0 0) 0x7ffbcc003520 con 0x7ffbe414be90 2024-03-25T19:28:39.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 >> v1:172.21.15.123:6790/0 conn(0x7ffbe40b1730 legacy=0x7ffbe40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 >> v1:172.21.15.123:6789/0 conn(0x7ffbe40b4d60 legacy=0x7ffbe413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 10 monclient: _finish_hunting 0 2024-03-25T19:28:39.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 1 monclient: found mon.noname-c 2024-03-25T19:28:39.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:39.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:28:39.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe40b1030 con 0x7ffbe414be90 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 10 monclient: _finish_auth 0 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 10 monclient: _check_auth_tickets 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ffbdc001a10 con 0x7ffbe414be90 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe159a640 10 monclient: dump: 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:39.964 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:39.965 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:39.965 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:39.965 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:39.965 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:39.965 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:39.965 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:39.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe8d3d640 5 monclient: authenticate success, global_id 4489 2024-03-25T19:28:39.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.962+0000 7ffbe8d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:39.971 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.971+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbdc001b70 con 0x7ffbe414be90 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.972+0000 7ffbe159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.972+0000 7ffbe159a640 1 -- 172.21.15.123:0/26337477 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ffbdc001f80 con 0x7ffbe414be90 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.972+0000 7ffbe159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.972+0000 7ffbe159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.972+0000 7ffbe159a640 10 monclient: dump: 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:39.972 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:39.973 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:39.973 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:39.973 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:39.973 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:39.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.972+0000 7ffbe8d3d640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:39.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.972+0000 7ffbe8d3d640 4 set_mon_vals no callback set 2024-03-25T19:28:39.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.972+0000 7ffbe8d3d640 10 monclient: shutdown 2024-03-25T19:28:39.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.972+0000 7ffbe8d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:39.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.972+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/26337477 >> v1:172.21.15.151:6789/0 conn(0x7ffbe414be90 legacy=0x7ffbe414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.973+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/26337477 shutdown_connections 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.973+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/26337477 >> 172.21.15.123:0/26337477 conn(0x7ffbe40617a0 msgr2=0x7ffbe4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.973+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/26337477 shutdown_connections 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.973+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/26337477 wait complete. 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.974+0000 7ffbe8d3d640 10 monclient: build_initial_monmap 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.974+0000 7ffbe8d3d640 10 monclient: monmap: 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:28:39.976 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.974+0000 7ffbe8d3d640 1 librados: starting msgr at 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.974+0000 7ffbe8d3d640 1 librados: starting objecter 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.974+0000 7ffbe8d3d640 1 Processor -- start 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.974+0000 7ffbe8d3d640 1 -- start start 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.974+0000 7ffbe8d3d640 1 librados: setting wanted keys 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.974+0000 7ffbe8d3d640 1 librados: calling monclient init 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.974+0000 7ffbe8d3d640 10 monclient: init 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.975+0000 7ffbe8d3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.975+0000 7ffbe8d3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.975+0000 7ffbe8d3d640 10 monclient(hunting): picked mon.noname-c con 0x7ffbe40b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:39.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.975+0000 7ffbe8d3d640 10 monclient(hunting): picked mon.noname-b con 0x7ffbe40b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:39.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.975+0000 7ffbe8d3d640 10 monclient(hunting): picked mon.noname-a con 0x7ffbe414be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:39.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.975+0000 7ffbe8d3d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe4135ee0 con 0x7ffbe414be90 2024-03-25T19:28:39.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.975+0000 7ffbe8d3d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe4135b00 con 0x7ffbe40b4d60 2024-03-25T19:28:39.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.975+0000 7ffbe8d3d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe4135c90 con 0x7ffbe40b1730 2024-03-25T19:28:39.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.975+0000 7ffbe8d3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:39.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.975+0000 7ffbe8d3d640 10 monclient(hunting): authenticate will time out at 5957.335198s 2024-03-25T19:28:39.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.977+0000 7ffbe1d9b640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ffbe40b4d60 0x7ffbe41d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:57344/0 (socket says 172.21.15.123:57344) 2024-03-25T19:28:39.978 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.977+0000 7ffbe1d9b640 1 -- 172.21.15.123:0/4233807165 learned_addr learned my addr 172.21.15.123:0/4233807165 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:39.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.981+0000 7ffbd2ffd640 10 client.?.objecter ms_handle_connect 0x7ffbe40b4d60 2024-03-25T19:28:39.981 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.981+0000 7ffbd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:39.985 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.985+0000 7ffbd2ffd640 10 client.?.objecter ms_handle_connect 0x7ffbe414be90 2024-03-25T19:28:39.985 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.985+0000 7ffbd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:39.987 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.987+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708050434 0 0) 0x7ffbe4135b00 con 0x7ffbe40b4d60 2024-03-25T19:28:39.987 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.987+0000 7ffbd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.987 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.987+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.987 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.987+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.987 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.987+0000 7ffbd2ffd640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:28:39.987 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.987+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc4004eb0 con 0x7ffbe40b4d60 2024-03-25T19:28:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.996+0000 7ffbd2ffd640 10 client.?.objecter ms_handle_connect 0x7ffbe40b1730 2024-03-25T19:28:39.996 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:39.996+0000 7ffbd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:40.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.001+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157138114 0 0) 0x7ffbe4135c90 con 0x7ffbe40b1730 2024-03-25T19:28:40.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.001+0000 7ffbd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:40.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.001+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:40.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.001+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:40.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.001+0000 7ffbd2ffd640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:28:40.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.001+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbe4135b00 con 0x7ffbe40b1730 2024-03-25T19:28:40.004 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.004+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1722776330 0 0) 0x7ffbc4004eb0 con 0x7ffbe40b4d60 2024-03-25T19:28:40.004 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.004+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbe4135c90 con 0x7ffbe40b4d60 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.004+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ffbd8001a10 con 0x7ffbe40b4d60 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.004+0000 7ffbd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.004+0000 7ffbd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.004+0000 7ffbd2ffd640 10 monclient(hunting): dump: 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:40.005 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:40.006 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:40.006 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:40.006 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:40.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.004+0000 7ffbd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:40.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.007+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047867772 0 0) 0x7ffbe4135ee0 con 0x7ffbe414be90 2024-03-25T19:28:40.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.007+0000 7ffbd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:40.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.007+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:40.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.007+0000 7ffbd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:40.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.007+0000 7ffbd2ffd640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:28:40.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.007+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc4004eb0 con 0x7ffbe414be90 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1180250384 0 0) 0x7ffbe4135c90 con 0x7ffbe40b4d60 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 >> v1:172.21.15.123:6790/0 conn(0x7ffbe40b1730 legacy=0x7ffbe412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 >> v1:172.21.15.123:6789/0 conn(0x7ffbe414be90 legacy=0x7ffbe41d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 1 monclient: found mon.b 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe4136960 con 0x7ffbe40b4d60 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbe8d3d640 5 monclient: authenticate success, global_id 4492 2024-03-25T19:28:40.013 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbe8d3d640 10 monclient: _renew_subs 2024-03-25T19:28:40.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbe8d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:40.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/4233807165 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe4130470 con 0x7ffbe40b4d60 2024-03-25T19:28:40.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbe8d3d640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:40.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbe8d3d640 10 monclient: _renew_subs 2024-03-25T19:28:40.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbe8d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:40.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/4233807165 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe413e690 con 0x7ffbe40b4d60 2024-03-25T19:28:40.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.012+0000 7ffbe8d3d640 1 librados: init done 2024-03-25T19:28:40.015 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.014+0000 7ffbe8d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:40.015 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.014+0000 7ffbe8d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:40.015 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.014+0000 7ffbe8d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:40.015 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.014+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/4233807165 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe4136b50 con 0x7ffbe40b4d60 2024-03-25T19:28:40.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.308+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbd8001c50 con 0x7ffbe40b4d60 2024-03-25T19:28:40.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.308+0000 7ffbd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:40.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.308+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7ffbd80038c0 con 0x7ffbe40b4d60 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.308+0000 7ffbd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.308+0000 7ffbd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.308+0000 7ffbd2ffd640 10 monclient: dump: 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:40.309 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:40.310 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.308+0000 7ffbe0d99640 4 set_mon_vals no callback set 2024-03-25T19:28:40.310 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.309+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7ffbd8003ae0 con 0x7ffbe40b4d60 2024-03-25T19:28:40.310 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.310+0000 7ffbe259c640 10 monclient: get_auth_request con 0x7ffbc406de00 auth_method 0 2024-03-25T19:28:40.310 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.310+0000 7ffbd2ffd640 10 client.4492.objecter ms_handle_connect 0x7ffbc406de00 2024-03-25T19:28:40.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.314+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7ffbd80546f0 con 0x7ffbe40b4d60 2024-03-25T19:28:40.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.314+0000 7ffbd2ffd640 10 client.4492.objecter ms_dispatch 0x7ffbe4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:40.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.314+0000 7ffbd2ffd640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:40.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.314+0000 7ffbd2ffd640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:40.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.314+0000 7ffbd2ffd640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-25T19:28:40.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.315+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffbd8050200 con 0x7ffbe40b4d60 2024-03-25T19:28:40.315 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.315+0000 7ffbd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:40.316 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.315+0000 7ffbd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:40.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.558+0000 7ffbe8d3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:40.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.558+0000 7ffbe8d3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:40.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.558+0000 7ffbe8d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:40.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.558+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/4233807165 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbe40627b0 con 0x7ffbe40b4d60 2024-03-25T19:28:40.584 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.584+0000 7ffbd2ffd640 1 -- 172.21.15.123:0/4233807165 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7ffbd8052130 con 0x7ffbe40b4d60 2024-03-25T19:28:40.584 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.584+0000 7ffbd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:40.584 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.584+0000 7ffbd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:40.585 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:28:40.585 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:40.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 10 librados: watch_flush enter 2024-03-25T19:28:40.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 10 librados: watch_flush exit 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 10 client.4492.objecter successfully canceled tick 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/4233807165 >> v1:172.21.15.151:6800/727684239 conn(0x7ffbc406de00 legacy=0x7ffbc40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 10 monclient: shutdown 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/4233807165 >> v1:172.21.15.151:6789/0 conn(0x7ffbe40b4d60 legacy=0x7ffbe41d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/4233807165 shutdown_connections 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/4233807165 >> 172.21.15.123:0/4233807165 conn(0x7ffbe40617a0 msgr2=0x7ffbe40b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/4233807165 shutdown_connections 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 1 -- 172.21.15.123:0/4233807165 wait complete. 2024-03-25T19:28:40.588 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:40.587+0000 7ffbe8d3d640 1 librados: shutdown 2024-03-25T19:28:40.599 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:47.406 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.521+0000 7f6fc475c640 10 monclient: get_monmap_and_config 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.521+0000 7f6fc475c640 10 monclient: build_initial_monmap 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.521+0000 7f6fc475c640 10 monclient: monmap: 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:28:47.521956+0000 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:28:47.521956+0000 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:47.521 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:28:47.522 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:28:47.522 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.521+0000 7f6fc475c640 1 Processor -- start 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.521+0000 7f6fc475c640 1 -- start start 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.521+0000 7f6fc475c640 10 monclient: init 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.522+0000 7f6fc475c640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.522+0000 7f6fc475c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.522+0000 7f6fc475c640 10 monclient(hunting): picked mon.noname-c con 0x7f6fbc0a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.522+0000 7f6fc475c640 10 monclient(hunting): picked mon.noname-b con 0x7f6fbc13f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.522+0000 7f6fc475c640 10 monclient(hunting): picked mon.noname-a con 0x7f6fbc1430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.522+0000 7f6fc475c640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc142520 con 0x7f6fbc1430c0 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.522+0000 7f6fc475c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc0b28a0 con 0x7f6fbc0a98f0 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.522+0000 7f6fc475c640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc0b2a30 con 0x7f6fbc13f710 2024-03-25T19:28:47.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.522+0000 7f6fc475c640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:47.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.522+0000 7f6fc475c640 10 monclient(hunting): authenticate will time out at 5964.882678s 2024-03-25T19:28:47.530 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.530+0000 7f6fc24f8640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f6fbc0a98f0 0x7f6fbc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:48828/0 (socket says 172.21.15.123:48828) 2024-03-25T19:28:47.530 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.530+0000 7f6fc24f8640 1 -- 172.21.15.123:0/3711232888 learned_addr learned my addr 172.21.15.123:0/3711232888 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:47.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.538+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463066055 0 0) 0x7f6fbc0b28a0 con 0x7f6fbc0a98f0 2024-03-25T19:28:47.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.538+0000 7f6fc14f6640 10 monclient(hunting): _negotiate 2024-03-25T19:28:47.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.538+0000 7f6fc14f6640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:47.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.538+0000 7f6fc14f6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:47.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.538+0000 7f6fc14f6640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:28:47.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.538+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa8003520 con 0x7f6fbc0a98f0 2024-03-25T19:28:47.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.541+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177265708 0 0) 0x7f6fbc142520 con 0x7f6fbc1430c0 2024-03-25T19:28:47.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.541+0000 7f6fc14f6640 10 monclient(hunting): _negotiate 2024-03-25T19:28:47.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.541+0000 7f6fc14f6640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:47.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.541+0000 7f6fc14f6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:47.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.541+0000 7f6fc14f6640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:28:47.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.541+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc0b28a0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3274558566 0 0) 0x7f6fbc0b28a0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 >> v1:172.21.15.123:6790/0 conn(0x7f6fbc13f710 legacy=0x7f6fbc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 >> v1:172.21.15.151:6789/0 conn(0x7f6fbc0a98f0 legacy=0x7f6fbc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 10 monclient: _finish_hunting 0 2024-03-25T19:28:47.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 1 monclient: found mon.noname-a 2024-03-25T19:28:47.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:47.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:28:47.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc0b44c0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 10 monclient: _finish_auth 0 2024-03-25T19:28:47.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 10 monclient: _check_auth_tickets 2024-03-25T19:28:47.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:47.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6fb40019f0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:47.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:47.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc14f6640 10 monclient: dump: 2024-03-25T19:28:47.553 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:47.553 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:47.553 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:47.554 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:47.554 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:47.554 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:47.554 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:47.554 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:47.554 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:47.554 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:47.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc475c640 5 monclient: authenticate success, global_id 4485 2024-03-25T19:28:47.554 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.552+0000 7f6fc475c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:47.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.564+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fb4001b50 con 0x7f6fbc1430c0 2024-03-25T19:28:47.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.564+0000 7f6fc14f6640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:47.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.564+0000 7f6fc14f6640 1 -- 172.21.15.123:0/3711232888 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6fb4001ee0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.564+0000 7f6fc475c640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:47.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.564+0000 7f6fc475c640 4 set_mon_vals no callback set 2024-03-25T19:28:47.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.564+0000 7f6fc14f6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.564+0000 7f6fc14f6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.564+0000 7f6fc14f6640 10 monclient: dump: 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:47.565 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:47.566 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.565+0000 7f6fc475c640 10 monclient: shutdown 2024-03-25T19:28:47.566 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.565+0000 7f6fc475c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.565+0000 7f6fc475c640 1 -- 172.21.15.123:0/3711232888 >> v1:172.21.15.123:6789/0 conn(0x7f6fbc1430c0 legacy=0x7f6fbc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.565+0000 7f6fc475c640 1 -- 172.21.15.123:0/3711232888 shutdown_connections 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.566+0000 7f6fc475c640 1 -- 172.21.15.123:0/3711232888 >> 172.21.15.123:0/3711232888 conn(0x7f6fbc0b7ea0 msgr2=0x7f6fbc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.566+0000 7f6fc475c640 1 -- 172.21.15.123:0/3711232888 shutdown_connections 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.566+0000 7f6fc475c640 1 -- 172.21.15.123:0/3711232888 wait complete. 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.567+0000 7f6fc475c640 10 monclient: build_initial_monmap 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.567+0000 7f6fc475c640 10 monclient: monmap: 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:28:47.567 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:47.568 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:28:47.568 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:28:47.568 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:47.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.567+0000 7f6fc475c640 1 librados: starting msgr at 2024-03-25T19:28:47.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.567+0000 7f6fc475c640 1 librados: starting objecter 2024-03-25T19:28:47.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.567+0000 7f6fc475c640 1 Processor -- start 2024-03-25T19:28:47.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.567+0000 7f6fc475c640 1 -- start start 2024-03-25T19:28:47.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.567+0000 7f6fc475c640 1 librados: setting wanted keys 2024-03-25T19:28:47.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.567+0000 7f6fc475c640 1 librados: calling monclient init 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.567+0000 7f6fc475c640 10 monclient: init 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.568+0000 7f6fc475c640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.568+0000 7f6fc475c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.568+0000 7f6fc475c640 10 monclient(hunting): picked mon.noname-c con 0x7f6fbc0a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.568+0000 7f6fc475c640 10 monclient(hunting): picked mon.noname-a con 0x7f6fbc13f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.568+0000 7f6fc475c640 10 monclient(hunting): picked mon.noname-b con 0x7f6fbc1430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.568+0000 7f6fc475c640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc0713f0 con 0x7f6fbc13f710 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.568+0000 7f6fc475c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc070020 con 0x7f6fbc1430c0 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.568+0000 7f6fc475c640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc070300 con 0x7f6fbc0a98f0 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.568+0000 7f6fc475c640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:47.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.568+0000 7f6fc475c640 10 monclient(hunting): authenticate will time out at 5964.928395s 2024-03-25T19:28:47.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.577+0000 7f6fc2cf9640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f6fbc1430c0 0x7f6fbc1dbdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:48860/0 (socket says 172.21.15.123:48860) 2024-03-25T19:28:47.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.577+0000 7f6fc2cf9640 1 -- 172.21.15.123:0/1645799524 learned_addr learned my addr 172.21.15.123:0/1645799524 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:47.578 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.578+0000 7f6fb2ffd640 10 client.?.objecter ms_handle_connect 0x7f6fbc1430c0 2024-03-25T19:28:47.578 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.578+0000 7f6fb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:47.579 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.579+0000 7f6fb2ffd640 10 client.?.objecter ms_handle_connect 0x7f6fbc13f710 2024-03-25T19:28:47.579 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.579+0000 7f6fb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:47.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.583+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064679484 0 0) 0x7f6fbc070020 con 0x7f6fbc1430c0 2024-03-25T19:28:47.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.583+0000 7f6fb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:47.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.583+0000 7f6fb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:47.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.583+0000 7f6fb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:47.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.583+0000 7f6fb2ffd640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:28:47.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.583+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa0004eb0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.587+0000 7f6fb2ffd640 10 client.?.objecter ms_handle_connect 0x7f6fbc0a98f0 2024-03-25T19:28:47.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.587+0000 7f6fb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:47.590 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.590+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746134819 0 0) 0x7f6fbc0713f0 con 0x7f6fbc13f710 2024-03-25T19:28:47.590 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.590+0000 7f6fb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:47.590 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.590+0000 7f6fb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:47.590 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.590+0000 7f6fb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:47.590 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.590+0000 7f6fb2ffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:28:47.590 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.590+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc070020 con 0x7f6fbc13f710 2024-03-25T19:28:47.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.592+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 262923373 0 0) 0x7f6fa0004eb0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.592+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fbc0713f0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.592+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6fb4001f60 con 0x7f6fbc1430c0 2024-03-25T19:28:47.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.592+0000 7f6fb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:47.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.592+0000 7f6fb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:47.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.592+0000 7f6fb2ffd640 10 monclient(hunting): dump: 2024-03-25T19:28:47.593 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:47.593 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:47.593 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:47.594 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:47.594 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:47.594 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:47.594 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:47.594 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:47.594 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:47.594 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:47.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.592+0000 7f6fb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 955822824 0 0) 0x7f6fbc0713f0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 >> v1:172.21.15.123:6790/0 conn(0x7f6fbc0a98f0 legacy=0x7f6fbc070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 >> v1:172.21.15.123:6789/0 conn(0x7f6fbc13f710 legacy=0x7f6fbc1db6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 1 monclient: found mon.b 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc071db0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:47.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fc475c640 5 monclient: authenticate success, global_id 4498 2024-03-25T19:28:47.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fc475c640 10 monclient: _renew_subs 2024-03-25T19:28:47.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fc475c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:47.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fc475c640 1 -- 172.21.15.123:0/1645799524 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fbc072060 con 0x7f6fbc1430c0 2024-03-25T19:28:47.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fc475c640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:47.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fc475c640 10 monclient: _renew_subs 2024-03-25T19:28:47.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fc475c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:47.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fc475c640 1 -- 172.21.15.123:0/1645799524 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fbc1e72e0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.599+0000 7f6fc475c640 1 librados: init done 2024-03-25T19:28:47.601 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.600+0000 7f6fc475c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:47.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.600+0000 7f6fc475c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:47.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.600+0000 7f6fc475c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:47.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.600+0000 7f6fc475c640 1 -- 172.21.15.123:0/1645799524 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fbc0627b0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.905 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.905+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fb4003c20 con 0x7f6fbc1430c0 2024-03-25T19:28:47.905 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.905+0000 7f6fb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:47.905 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.905+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6fb4003fb0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.905 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.905+0000 7f6fb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.905+0000 7f6fb2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.905+0000 7f6fc0cf5640 4 set_mon_vals no callback set 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.905+0000 7f6fb2ffd640 10 monclient: dump: 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:47.906 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:47.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.906+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f6fb40041d0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.906+0000 7f6fc24f8640 10 monclient: get_auth_request con 0x7f6fa006dd10 auth_method 0 2024-03-25T19:28:47.907 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.907+0000 7f6fb2ffd640 10 client.4498.objecter ms_handle_connect 0x7f6fa006dd10 2024-03-25T19:28:47.910 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.910+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f6fb4054ad0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.910+0000 7f6fb2ffd640 10 client.4498.objecter ms_dispatch 0x7f6fbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:47.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.910+0000 7f6fb2ffd640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:47.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.910+0000 7f6fb2ffd640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:47.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.911+0000 7f6fb2ffd640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T19:28:47.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.911+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fb40505e0 con 0x7f6fbc1430c0 2024-03-25T19:28:47.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.911+0000 7f6fb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:47.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:47.911+0000 7f6fb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:48.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.157+0000 7f6fc475c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:48.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.157+0000 7f6fc475c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:48.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.157+0000 7f6fc475c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:28:48.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.157+0000 7f6fc475c640 1 -- 172.21.15.123:0/1645799524 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fbc069f40 con 0x7f6fbc1430c0 2024-03-25T19:28:48.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.185+0000 7f6fb2ffd640 1 -- 172.21.15.123:0/1645799524 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f6fb4052510 con 0x7f6fbc1430c0 2024-03-25T19:28:48.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.185+0000 7f6fb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:48.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.185+0000 7f6fb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:48.186 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:28:48.186 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:48.188 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.187+0000 7f6fc475c640 10 librados: watch_flush enter 2024-03-25T19:28:48.188 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 10 librados: watch_flush exit 2024-03-25T19:28:48.188 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T19:28:48.188 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 10 client.4498.objecter successfully canceled tick 2024-03-25T19:28:48.188 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 1 -- 172.21.15.123:0/1645799524 >> v1:172.21.15.151:6800/727684239 conn(0x7f6fa006dd10 legacy=0x7f6fa00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:48.189 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 10 monclient: shutdown 2024-03-25T19:28:48.189 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:48.189 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 1 -- 172.21.15.123:0/1645799524 >> v1:172.21.15.151:6789/0 conn(0x7f6fbc1430c0 legacy=0x7f6fbc1dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:48.189 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 1 -- 172.21.15.123:0/1645799524 shutdown_connections 2024-03-25T19:28:48.189 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 1 -- 172.21.15.123:0/1645799524 >> 172.21.15.123:0/1645799524 conn(0x7f6fbc0b7ea0 msgr2=0x7f6fbc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:48.189 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 1 -- 172.21.15.123:0/1645799524 shutdown_connections 2024-03-25T19:28:48.189 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 1 -- 172.21.15.123:0/1645799524 wait complete. 2024-03-25T19:28:48.189 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:48.188+0000 7f6fc475c640 1 librados: shutdown 2024-03-25T19:28:48.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:55.006 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:55.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.125+0000 7fe4f58ba640 10 monclient: get_monmap_and_config 2024-03-25T19:28:55.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.125+0000 7fe4f58ba640 10 monclient: build_initial_monmap 2024-03-25T19:28:55.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.125+0000 7fe4f58ba640 10 monclient: monmap: 2024-03-25T19:28:55.125 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:55.125 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:55.125 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:28:55.126195+0000 2024-03-25T19:28:55.126 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:28:55.126195+0000 2024-03-25T19:28:55.126 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:55.126 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:55.126 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:55.126 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:28:55.126 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:28:55.126 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:55.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.126+0000 7fe4f58ba640 1 Processor -- start 2024-03-25T19:28:55.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.126+0000 7fe4f58ba640 1 -- start start 2024-03-25T19:28:55.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.126+0000 7fe4f58ba640 10 monclient: init 2024-03-25T19:28:55.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.127+0000 7fe4f58ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:55.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.127+0000 7fe4f58ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:55.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.127+0000 7fe4f58ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe4f00a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:55.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.127+0000 7fe4f58ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe4f013f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:55.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.127+0000 7fe4f58ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe4f01430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:55.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.127+0000 7fe4f58ba640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f0142520 con 0x7fe4f01430c0 2024-03-25T19:28:55.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.127+0000 7fe4f58ba640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00b28a0 con 0x7fe4f00a98f0 2024-03-25T19:28:55.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.127+0000 7fe4f58ba640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00b2a30 con 0x7fe4f013f710 2024-03-25T19:28:55.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.127+0000 7fe4f58ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:55.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.127+0000 7fe4f58ba640 10 monclient(hunting): authenticate will time out at 5972.487315s 2024-03-25T19:28:55.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.131+0000 7fe4ef7fe640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe4f01430c0 0x7fe4f014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:39848/0 (socket says 172.21.15.123:39848) 2024-03-25T19:28:55.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.131+0000 7fe4ef7fe640 1 -- 172.21.15.123:0/4129377059 learned_addr learned my addr 172.21.15.123:0/4129377059 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:55.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.139+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291285024 0 0) 0x7fe4f0142520 con 0x7fe4f01430c0 2024-03-25T19:28:55.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.139+0000 7fe4ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:55.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.139+0000 7fe4ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:55.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.139+0000 7fe4ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:55.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.139+0000 7fe4ee7fc640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:28:55.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.139+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4dc003520 con 0x7fe4f01430c0 2024-03-25T19:28:55.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.159+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462994925 0 0) 0x7fe4f00b28a0 con 0x7fe4f00a98f0 2024-03-25T19:28:55.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.159+0000 7fe4ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:55.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.159+0000 7fe4ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:55.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.159+0000 7fe4ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:55.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.159+0000 7fe4ee7fc640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:28:55.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.159+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f0142520 con 0x7fe4f00a98f0 2024-03-25T19:28:55.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3950935652 0 0) 0x7fe4dc003520 con 0x7fe4f01430c0 2024-03-25T19:28:55.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 >> v1:172.21.15.123:6790/0 conn(0x7fe4f013f710 legacy=0x7fe4f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 >> v1:172.21.15.151:6789/0 conn(0x7fe4f00a98f0 legacy=0x7fe4f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 1 monclient: found mon.noname-a 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f00b44c0 con 0x7fe4f01430c0 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe4e8001a10 con 0x7fe4f01430c0 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4f58ba640 5 monclient: authenticate success, global_id 4491 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4f58ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:55.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4ee7fc640 10 monclient: dump: 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:55.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.160+0000 7fe4f58ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:55.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.175+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4e8001b70 con 0x7fe4f01430c0 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.175+0000 7fe4ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.175+0000 7fe4ee7fc640 1 -- 172.21.15.123:0/4129377059 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe4e8001f80 con 0x7fe4f01430c0 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.175+0000 7fe4f58ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.175+0000 7fe4f58ba640 4 set_mon_vals no callback set 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.175+0000 7fe4ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.175+0000 7fe4ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.175+0000 7fe4ee7fc640 10 monclient: dump: 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:55.175 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:55.176 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:55.176 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:55.176 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:55.176 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:55.176 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:55.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.175+0000 7fe4f58ba640 10 monclient: shutdown 2024-03-25T19:28:55.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.176+0000 7fe4f58ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:55.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.176+0000 7fe4f58ba640 1 -- 172.21.15.123:0/4129377059 >> v1:172.21.15.123:6789/0 conn(0x7fe4f01430c0 legacy=0x7fe4f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.176+0000 7fe4f58ba640 1 -- 172.21.15.123:0/4129377059 shutdown_connections 2024-03-25T19:28:55.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.176+0000 7fe4f58ba640 1 -- 172.21.15.123:0/4129377059 >> 172.21.15.123:0/4129377059 conn(0x7fe4f00b7ea0 msgr2=0x7fe4f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:55.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.176+0000 7fe4f58ba640 1 -- 172.21.15.123:0/4129377059 shutdown_connections 2024-03-25T19:28:55.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.176+0000 7fe4f58ba640 1 -- 172.21.15.123:0/4129377059 wait complete. 2024-03-25T19:28:55.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.176+0000 7fe4f58ba640 10 monclient: build_initial_monmap 2024-03-25T19:28:55.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.176+0000 7fe4f58ba640 10 monclient: monmap: 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 1 librados: starting msgr at 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 1 librados: starting objecter 2024-03-25T19:28:55.179 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 1 Processor -- start 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 1 -- start start 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 1 librados: setting wanted keys 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 1 librados: calling monclient init 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 10 monclient: init 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe4f00a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe4f013f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe4f01430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00abb00 con 0x7fe4f01430c0 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00abde0 con 0x7fe4f00a98f0 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f0138e20 con 0x7fe4f013f710 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:55.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.177+0000 7fe4f58ba640 10 monclient(hunting): authenticate will time out at 5972.537841s 2024-03-25T19:28:55.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.193+0000 7fe4ef7fe640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe4f01430c0 0x7fe4f00ac200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:39896/0 (socket says 172.21.15.123:39896) 2024-03-25T19:28:55.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.193+0000 7fe4ef7fe640 1 -- 172.21.15.123:0/135742159 learned_addr learned my addr 172.21.15.123:0/135742159 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:28:55.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.195+0000 7fe4ecff9640 10 client.?.objecter ms_handle_connect 0x7fe4f01430c0 2024-03-25T19:28:55.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.195+0000 7fe4ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:55.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.199+0000 7fe4ecff9640 10 client.?.objecter ms_handle_connect 0x7fe4f00a98f0 2024-03-25T19:28:55.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.199+0000 7fe4ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:55.201 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.201+0000 7fe4ecff9640 10 client.?.objecter ms_handle_connect 0x7fe4f013f710 2024-03-25T19:28:55.201 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.201+0000 7fe4ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:55.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.203+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860855297 0 0) 0x7fe4f00abb00 con 0x7fe4f01430c0 2024-03-25T19:28:55.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.203+0000 7fe4ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:55.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.203+0000 7fe4ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:55.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.203+0000 7fe4ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:55.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.203+0000 7fe4ecff9640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:28:55.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.203+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d0004eb0 con 0x7fe4f01430c0 2024-03-25T19:28:55.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.210+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309362056 0 0) 0x7fe4f0138e20 con 0x7fe4f013f710 2024-03-25T19:28:55.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.210+0000 7fe4ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:55.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.210+0000 7fe4ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:55.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.210+0000 7fe4ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:55.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.210+0000 7fe4ecff9640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:28:55.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.210+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f00abb00 con 0x7fe4f013f710 2024-03-25T19:28:55.211 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.211+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740854917 0 0) 0x7fe4f00abde0 con 0x7fe4f00a98f0 2024-03-25T19:28:55.211 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.211+0000 7fe4ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:55.211 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.211+0000 7fe4ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:55.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.211+0000 7fe4ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:55.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.211+0000 7fe4ecff9640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:28:55.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.211+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f0138e20 con 0x7fe4f00a98f0 2024-03-25T19:28:55.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.227+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4177158995 0 0) 0x7fe4d0004eb0 con 0x7fe4f01430c0 2024-03-25T19:28:55.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.227+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4f00abde0 con 0x7fe4f01430c0 2024-03-25T19:28:55.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.227+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe4e8001ec0 con 0x7fe4f01430c0 2024-03-25T19:28:55.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.227+0000 7fe4ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:55.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.227+0000 7fe4ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:55.227 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.227+0000 7fe4ecff9640 10 monclient(hunting): dump: 2024-03-25T19:28:55.227 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:55.227 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:55.227 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:55.228 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:55.228 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:55.228 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:55.228 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:55.228 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:55.228 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:55.228 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:55.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.227+0000 7fe4ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.229+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439891587 0 0) 0x7fe4f00abb00 con 0x7fe4f013f710 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.229+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d0004eb0 con 0x7fe4f013f710 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.229+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe4d80018b0 con 0x7fe4f013f710 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.229+0000 7fe4ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.229+0000 7fe4ecff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.229+0000 7fe4ecff9640 10 monclient(hunting): dump: 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:55.229 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:55.230 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:55.230 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:55.230 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:55.230 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:55.230 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:55.235 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1398938397 0 0) 0x7fe4f00abde0 con 0x7fe4f01430c0 2024-03-25T19:28:55.235 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 >> v1:172.21.15.123:6790/0 conn(0x7fe4f013f710 legacy=0x7fe4f0136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.235 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 >> v1:172.21.15.151:6789/0 conn(0x7fe4f00a98f0 legacy=0x7fe4f01385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 10 monclient: _finish_hunting 0 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 1 monclient: found mon.a 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f0137b00 con 0x7fe4f01430c0 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 10 monclient: _finish_auth 0 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 10 monclient: _check_auth_tickets 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4f58ba640 5 monclient: authenticate success, global_id 4494 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4f58ba640 10 monclient: _renew_subs 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4f58ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4f58ba640 1 -- 172.21.15.123:0/135742159 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f01e34b0 con 0x7fe4f01430c0 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4f58ba640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:55.236 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4f58ba640 10 monclient: _renew_subs 2024-03-25T19:28:55.237 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4f58ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:28:55.237 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4f58ba640 1 -- 172.21.15.123:0/135742159 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f01e3820 con 0x7fe4f01430c0 2024-03-25T19:28:55.237 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.235+0000 7fe4f58ba640 1 librados: init done 2024-03-25T19:28:55.237 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.236+0000 7fe4f58ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:55.237 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.236+0000 7fe4f58ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:55.237 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.236+0000 7fe4f58ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:28:55.237 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.236+0000 7fe4f58ba640 1 -- 172.21.15.123:0/135742159 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f00627b0 con 0x7fe4f01430c0 2024-03-25T19:28:55.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.549+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4e8002020 con 0x7fe4f01430c0 2024-03-25T19:28:55.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.549+0000 7fe4ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.549+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe4e8004fb0 con 0x7fe4f01430c0 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.549+0000 7fe4ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.549+0000 7fe4ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.549+0000 7fe4ecff9640 10 monclient: dump: 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:28:55.550 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:28:55.551 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:28:55.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.549+0000 7fe4edffb640 4 set_mon_vals no callback set 2024-03-25T19:28:55.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.550+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fe4e801af80 con 0x7fe4f01430c0 2024-03-25T19:28:55.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.551+0000 7fe4eeffd640 10 monclient: get_auth_request con 0x7fe4d006df70 auth_method 0 2024-03-25T19:28:55.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.551+0000 7fe4ecff9640 10 client.4494.objecter ms_handle_connect 0x7fe4d006df70 2024-03-25T19:28:55.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.556+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fe4e8054b30 con 0x7fe4f01430c0 2024-03-25T19:28:55.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.556+0000 7fe4ecff9640 10 client.4494.objecter ms_dispatch 0x7fe4f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:55.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.556+0000 7fe4ecff9640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:55.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.556+0000 7fe4ecff9640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:55.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.556+0000 7fe4ecff9640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-25T19:28:55.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.556+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4e8050470 con 0x7fe4f01430c0 2024-03-25T19:28:55.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.556+0000 7fe4ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:55.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.556+0000 7fe4ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:55.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.790+0000 7fe4f58ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:55.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.790+0000 7fe4f58ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:55.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.790+0000 7fe4f58ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:28:55.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.790+0000 7fe4f58ba640 1 -- 172.21.15.123:0/135742159 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4f00b1760 con 0x7fe4f01430c0 2024-03-25T19:28:55.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.856+0000 7fe4ecff9640 1 -- 172.21.15.123:0/135742159 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fe4e8052570 con 0x7fe4f01430c0 2024-03-25T19:28:55.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.856+0000 7fe4ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:55.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.856+0000 7fe4ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:55.857 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:28:55.857 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:55.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.858+0000 7fe4f58ba640 10 librados: watch_flush enter 2024-03-25T19:28:55.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 10 librados: watch_flush exit 2024-03-25T19:28:55.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-25T19:28:55.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 10 client.4494.objecter successfully canceled tick 2024-03-25T19:28:55.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 1 -- 172.21.15.123:0/135742159 >> v1:172.21.15.151:6800/727684239 conn(0x7fe4d006df70 legacy=0x7fe4d0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 10 monclient: shutdown 2024-03-25T19:28:55.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:55.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 1 -- 172.21.15.123:0/135742159 >> v1:172.21.15.123:6789/0 conn(0x7fe4f01430c0 legacy=0x7fe4f00ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 1 -- 172.21.15.123:0/135742159 shutdown_connections 2024-03-25T19:28:55.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 1 -- 172.21.15.123:0/135742159 >> 172.21.15.123:0/135742159 conn(0x7fe4f00b7ea0 msgr2=0x7fe4f014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:55.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 1 -- 172.21.15.123:0/135742159 shutdown_connections 2024-03-25T19:28:55.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 1 -- 172.21.15.123:0/135742159 wait complete. 2024-03-25T19:28:55.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:28:55.859+0000 7fe4f58ba640 1 librados: shutdown 2024-03-25T19:28:55.871 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:55.921 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:06:45.493 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:55.945 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:06:45.516 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:55.979 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:06:45.551 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:55.988 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:06:45.522 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:55.989 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:06:45.562 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:55.994 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:06:45.527 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:55.999 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:06:45.572 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:56.026 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:06:45.559 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:56.029 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:06:45.602 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:02.677 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:02.778 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.778+0000 7f947282a640 10 monclient: get_monmap_and_config 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.778+0000 7f947282a640 10 monclient: build_initial_monmap 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.778+0000 7f947282a640 10 monclient: monmap: 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:29:02.779869+0000 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:29:02.779869+0000 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:29:02.779 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:29:02.780 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.779+0000 7f947282a640 1 Processor -- start 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.779+0000 7f947282a640 1 -- start start 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.779+0000 7f947282a640 10 monclient: init 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f947282a640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f947282a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f947282a640 10 monclient(hunting): picked mon.noname-b con 0x7f946c0ae6e0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f947282a640 10 monclient(hunting): picked mon.noname-c con 0x7f946c06f080 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f947282a640 10 monclient(hunting): picked mon.noname-a con 0x7f946c0704e0 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f947282a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0718d0 con 0x7f946c0704e0 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f947282a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c06fa30 con 0x7f946c06f080 2024-03-25T19:29:02.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f947282a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c070090 con 0x7f946c0ae6e0 2024-03-25T19:29:02.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f947282a640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:02.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f947282a640 10 monclient(hunting): authenticate will time out at 5980.140203s 2024-03-25T19:29:02.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f946bfff640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f946c0ae6e0 0x7f946c13c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:51586/0 (socket says 172.21.15.123:51586) 2024-03-25T19:29:02.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.780+0000 7f946bfff640 1 -- 172.21.15.123:0/1370586938 learned_addr learned my addr 172.21.15.123:0/1370586938 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:02.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.817+0000 7f946affd640 1 -- 172.21.15.123:0/1370586938 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771204573 0 0) 0x7f946c0718d0 con 0x7f946c0704e0 2024-03-25T19:29:02.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.817+0000 7f946affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:02.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.817+0000 7f946affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:02.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.817+0000 7f946affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:02.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.817+0000 7f946affd640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:29:02.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.817+0000 7f946affd640 1 -- 172.21.15.123:0/1370586938 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9454003520 con 0x7f946c0704e0 2024-03-25T19:29:02.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.837+0000 7f946affd640 1 -- 172.21.15.123:0/1370586938 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1090179808 0 0) 0x7f9454003520 con 0x7f946c0704e0 2024-03-25T19:29:02.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.837+0000 7f946affd640 1 -- 172.21.15.123:0/1370586938 >> v1:172.21.15.123:6790/0 conn(0x7f946c0ae6e0 legacy=0x7f946c13c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 1 -- 172.21.15.123:0/1370586938 >> v1:172.21.15.151:6789/0 conn(0x7f946c06f080 legacy=0x7f946c071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:02.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 1 monclient: found mon.noname-a 2024-03-25T19:29:02.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:02.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:29:02.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 1 -- 172.21.15.123:0/1370586938 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c06f510 con 0x7f946c0704e0 2024-03-25T19:29:02.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 10 monclient: _finish_auth 0 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 1 -- 172.21.15.123:0/1370586938 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9464001a10 con 0x7f946c0704e0 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f946affd640 10 monclient: dump: 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:02.839 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:02.840 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:02.840 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:02.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f947282a640 5 monclient: authenticate success, global_id 4497 2024-03-25T19:29:02.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.838+0000 7f947282a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:02.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.864+0000 7f946affd640 1 -- 172.21.15.123:0/1370586938 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9464001b70 con 0x7f946c0704e0 2024-03-25T19:29:02.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.864+0000 7f946affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:02.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.864+0000 7f946affd640 1 -- 172.21.15.123:0/1370586938 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9464001f80 con 0x7f946c0704e0 2024-03-25T19:29:02.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.864+0000 7f947282a640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.864+0000 7f947282a640 4 set_mon_vals no callback set 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.864+0000 7f946affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.864+0000 7f946affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.864+0000 7f946affd640 10 monclient: dump: 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:02.865 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:02.867 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.865+0000 7f947282a640 10 monclient: shutdown 2024-03-25T19:29:02.867 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.865+0000 7f947282a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:02.867 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.865+0000 7f947282a640 1 -- 172.21.15.123:0/1370586938 >> v1:172.21.15.123:6789/0 conn(0x7f946c0704e0 legacy=0x7f946c14a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.867 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.865+0000 7f947282a640 1 -- 172.21.15.123:0/1370586938 shutdown_connections 2024-03-25T19:29:02.867 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.865+0000 7f947282a640 1 -- 172.21.15.123:0/1370586938 >> 172.21.15.123:0/1370586938 conn(0x7f946c06b630 msgr2=0x7f946c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:02.867 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.865+0000 7f947282a640 1 -- 172.21.15.123:0/1370586938 shutdown_connections 2024-03-25T19:29:02.867 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.865+0000 7f947282a640 1 -- 172.21.15.123:0/1370586938 wait complete. 2024-03-25T19:29:02.867 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.866+0000 7f947282a640 10 monclient: build_initial_monmap 2024-03-25T19:29:02.867 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.866+0000 7f947282a640 10 monclient: monmap: 2024-03-25T19:29:02.867 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.867+0000 7f947282a640 1 librados: starting msgr at 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.867+0000 7f947282a640 1 librados: starting objecter 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.867+0000 7f947282a640 1 Processor -- start 2024-03-25T19:29:02.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.867+0000 7f947282a640 1 -- start start 2024-03-25T19:29:02.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.867+0000 7f947282a640 1 librados: setting wanted keys 2024-03-25T19:29:02.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.867+0000 7f947282a640 1 librados: calling monclient init 2024-03-25T19:29:02.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.867+0000 7f947282a640 10 monclient: init 2024-03-25T19:29:02.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.868+0000 7f947282a640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:02.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.868+0000 7f947282a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:02.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.868+0000 7f947282a640 10 monclient(hunting): picked mon.noname-a con 0x7f946c0704e0 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:02.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.868+0000 7f947282a640 10 monclient(hunting): picked mon.noname-b con 0x7f946c0ae6e0 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:02.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.868+0000 7f947282a640 10 monclient(hunting): picked mon.noname-c con 0x7f946c1373f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:02.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.868+0000 7f947282a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0ac370 con 0x7f946c0704e0 2024-03-25T19:29:02.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.868+0000 7f947282a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0ab1e0 con 0x7f946c0ae6e0 2024-03-25T19:29:02.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.868+0000 7f947282a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0ab370 con 0x7f946c1373f0 2024-03-25T19:29:02.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.868+0000 7f947282a640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:02.871 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.868+0000 7f947282a640 10 monclient(hunting): authenticate will time out at 5980.228173s 2024-03-25T19:29:02.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.874+0000 7f946b7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f946c0ae6e0 0x7f946c136ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:54570/0 (socket says 172.21.15.123:54570) 2024-03-25T19:29:02.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.874+0000 7f946b7fe640 1 -- 172.21.15.123:0/1324950302 learned_addr learned my addr 172.21.15.123:0/1324950302 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:02.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.877+0000 7f9468ff9640 10 client.?.objecter ms_handle_connect 0x7f946c0ae6e0 2024-03-25T19:29:02.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.877+0000 7f9468ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:02.882 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.883+0000 7f9468ff9640 10 client.?.objecter ms_handle_connect 0x7f946c0704e0 2024-03-25T19:29:02.882 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.883+0000 7f9468ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:02.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.892+0000 7f9468ff9640 10 client.?.objecter ms_handle_connect 0x7f946c1373f0 2024-03-25T19:29:02.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.892+0000 7f9468ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:02.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.912+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756796088 0 0) 0x7f946c0ac370 con 0x7f946c0704e0 2024-03-25T19:29:02.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.912+0000 7f9468ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:02.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.912+0000 7f9468ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:02.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.912+0000 7f9468ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:02.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.912+0000 7f9468ff9640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:29:02.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.912+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9458004eb0 con 0x7f946c0704e0 2024-03-25T19:29:02.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.913+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36817445 0 0) 0x7f946c0ab1e0 con 0x7f946c0ae6e0 2024-03-25T19:29:02.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.913+0000 7f9468ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:02.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.913+0000 7f9468ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:02.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.913+0000 7f9468ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:02.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.913+0000 7f9468ff9640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:29:02.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.913+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946c0ac370 con 0x7f946c0ae6e0 2024-03-25T19:29:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.916+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404194652 0 0) 0x7f946c0ab370 con 0x7f946c1373f0 2024-03-25T19:29:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.916+0000 7f9468ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.916+0000 7f9468ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.916+0000 7f9468ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.916+0000 7f9468ff9640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:29:02.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.916+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946c0ab1e0 con 0x7f946c1373f0 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.928+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 389637709 0 0) 0x7f9458004eb0 con 0x7f946c0704e0 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.928+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f946c0ab370 con 0x7f946c0704e0 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.929+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f945c001800 con 0x7f946c0704e0 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.929+0000 7f9468ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.929+0000 7f9468ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.929+0000 7f9468ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:02.929 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:02.930 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:02.930 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:02.930 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:02.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.929+0000 7f9468ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:02.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.932+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 816743833 0 0) 0x7f946c0ab1e0 con 0x7f946c1373f0 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.932+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9458004eb0 con 0x7f946c1373f0 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.932+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9464001e80 con 0x7f946c1373f0 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.932+0000 7f9468ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.932+0000 7f9468ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.932+0000 7f9468ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:02.933 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:02.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.937+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4103458715 0 0) 0x7f946c0ac370 con 0x7f946c0ae6e0 2024-03-25T19:29:02.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.937+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f946c0ab1e0 con 0x7f946c0ae6e0 2024-03-25T19:29:02.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.937+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f94600018f0 con 0x7f946c0ae6e0 2024-03-25T19:29:02.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.937+0000 7f9468ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.937+0000 7f9468ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:02.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.937+0000 7f9468ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:02.937 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:02.937 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:02.937 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:02.937 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:02.938 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.938 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:02.938 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:02.938 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:02.938 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:02.938 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:02.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.944+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2800840635 0 0) 0x7f9458004eb0 con 0x7f946c1373f0 2024-03-25T19:29:02.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 >> v1:172.21.15.151:6789/0 conn(0x7f946c0ae6e0 legacy=0x7f946c136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 >> v1:172.21.15.123:6789/0 conn(0x7f946c0704e0 legacy=0x7f946c0ab730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f9468ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:02.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f9468ff9640 1 monclient: found mon.c 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f9468ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f9468ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c0ab500 con 0x7f946c1373f0 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f9468ff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f9468ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f9468ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f947282a640 5 monclient: authenticate success, global_id 4532 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f947282a640 10 monclient: _renew_subs 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f947282a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f947282a640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c142790 con 0x7f946c1373f0 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f947282a640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f947282a640 10 monclient: _renew_subs 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f947282a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f947282a640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c1d5a60 con 0x7f946c1373f0 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.945+0000 7f947282a640 1 librados: init done 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.946+0000 7f947282a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.946+0000 7f947282a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.946+0000 7f947282a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:02.946+0000 7f947282a640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c0627b0 con 0x7f946c1373f0 2024-03-25T19:29:03.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.249+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9464003c20 con 0x7f946c1373f0 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.249+0000 7f9468ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.249+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f94640015d0 con 0x7f946c1373f0 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.249+0000 7f9468ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.249+0000 7f9468ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.249+0000 7f9468ff9640 10 monclient: dump: 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:03.250 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:03.251 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:03.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.249+0000 7f946a7fc640 4 set_mon_vals no callback set 2024-03-25T19:29:03.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.250+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f9464019da0 con 0x7f946c1373f0 2024-03-25T19:29:03.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.251+0000 7f946bfff640 10 monclient: get_auth_request con 0x7f945806e110 auth_method 0 2024-03-25T19:29:03.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.252+0000 7f9468ff9640 10 client.4532.objecter ms_handle_connect 0x7f945806e110 2024-03-25T19:29:03.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.258+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f94640539e0 con 0x7f946c1373f0 2024-03-25T19:29:03.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.258+0000 7f9468ff9640 10 client.4532.objecter ms_dispatch 0x7f946c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:03.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.258+0000 7f9468ff9640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:03.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.258+0000 7f9468ff9640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:03.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.258+0000 7f9468ff9640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-25T19:29:03.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.258+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f946404f320 con 0x7f946c1373f0 2024-03-25T19:29:03.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.258+0000 7f9468ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:03.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.258+0000 7f9468ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:03.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.503+0000 7f947282a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:03.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.503+0000 7f947282a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:03.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.503+0000 7f947282a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:03.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.503+0000 7f947282a640 1 -- 172.21.15.123:0/1324950302 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f946c074d80 con 0x7f946c1373f0 2024-03-25T19:29:03.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.534+0000 7f9468ff9640 1 -- 172.21.15.123:0/1324950302 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f946c074d80 con 0x7f946c1373f0 2024-03-25T19:29:03.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.535+0000 7f9468ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:03.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.535+0000 7f9468ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:03.535 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:29:03.535 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:03.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 10 librados: watch_flush enter 2024-03-25T19:29:03.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 10 librados: watch_flush exit 2024-03-25T19:29:03.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-25T19:29:03.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 10 client.4532.objecter successfully canceled tick 2024-03-25T19:29:03.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 1 -- 172.21.15.123:0/1324950302 >> v1:172.21.15.151:6800/727684239 conn(0x7f945806e110 legacy=0x7f94580705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:03.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 10 monclient: shutdown 2024-03-25T19:29:03.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:03.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 1 -- 172.21.15.123:0/1324950302 >> v1:172.21.15.123:6790/0 conn(0x7f946c1373f0 legacy=0x7f946c1d5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:03.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 1 -- 172.21.15.123:0/1324950302 shutdown_connections 2024-03-25T19:29:03.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 1 -- 172.21.15.123:0/1324950302 >> 172.21.15.123:0/1324950302 conn(0x7f946c06b630 msgr2=0x7f946c13c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:03.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 1 -- 172.21.15.123:0/1324950302 shutdown_connections 2024-03-25T19:29:03.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 1 -- 172.21.15.123:0/1324950302 wait complete. 2024-03-25T19:29:03.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:03.537+0000 7f947282a640 1 librados: shutdown 2024-03-25T19:29:03.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:10.356 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:10.468 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.468+0000 7f7578e31640 10 monclient: get_monmap_and_config 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.468+0000 7f7578e31640 10 monclient: build_initial_monmap 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.468+0000 7f7578e31640 10 monclient: monmap: 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:29:10.469596+0000 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:29:10.469596+0000 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:29:10.469 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:10.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.469+0000 7f7578e31640 1 Processor -- start 2024-03-25T19:29:10.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.469+0000 7f7578e31640 1 -- start start 2024-03-25T19:29:10.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.469+0000 7f7578e31640 10 monclient: init 2024-03-25T19:29:10.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.470+0000 7f7578e31640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:10.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.470+0000 7f7578e31640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:10.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.470+0000 7f7578e31640 10 monclient(hunting): picked mon.noname-b con 0x7f75740a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:10.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.470+0000 7f7578e31640 10 monclient(hunting): picked mon.noname-c con 0x7f757413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:10.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.470+0000 7f7578e31640 10 monclient(hunting): picked mon.noname-a con 0x7f75741430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:10.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.470+0000 7f7578e31640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7574142520 con 0x7f75741430c0 2024-03-25T19:29:10.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.470+0000 7f7578e31640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740b28a0 con 0x7f757413f710 2024-03-25T19:29:10.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.470+0000 7f7578e31640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740b2a30 con 0x7f75740a98f0 2024-03-25T19:29:10.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.470+0000 7f7578e31640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:10.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.470+0000 7f7578e31640 10 monclient(hunting): authenticate will time out at 5987.830162s 2024-03-25T19:29:10.479 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.479+0000 7f756bfff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f757413f710 0x7f75741429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:51198/0 (socket says 172.21.15.123:51198) 2024-03-25T19:29:10.479 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.479+0000 7f756bfff640 1 -- 172.21.15.123:0/717951677 learned_addr learned my addr 172.21.15.123:0/717951677 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:10.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.496+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887166139 0 0) 0x7f75740b2a30 con 0x7f75740a98f0 2024-03-25T19:29:10.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.496+0000 7f7571d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:10.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.496+0000 7f7571d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:10.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.496+0000 7f7571d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:10.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.496+0000 7f7571d9b640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:29:10.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.496+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7560003520 con 0x7f75740a98f0 2024-03-25T19:29:10.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.499+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659883852 0 0) 0x7f75740b28a0 con 0x7f757413f710 2024-03-25T19:29:10.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.499+0000 7f7571d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:10.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.499+0000 7f7571d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:10.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.499+0000 7f7571d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:10.500 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.499+0000 7f7571d9b640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:29:10.500 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.499+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75740b2a30 con 0x7f757413f710 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2546735755 0 0) 0x7f7560003520 con 0x7f75740a98f0 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 >> v1:172.21.15.151:6789/0 conn(0x7f757413f710 legacy=0x7f75741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 >> v1:172.21.15.123:6789/0 conn(0x7f75741430c0 legacy=0x7f757414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 1 monclient: found mon.noname-b 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75740b44c0 con 0x7f75740a98f0 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 10 monclient: _finish_auth 0 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7564001690 con 0x7f75740a98f0 2024-03-25T19:29:10.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7571d9b640 10 monclient: dump: 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7578e31640 5 monclient: authenticate success, global_id 4535 2024-03-25T19:29:10.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.502+0000 7f7578e31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.515+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75640017f0 con 0x7f75740a98f0 2024-03-25T19:29:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.515+0000 7f7571d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.515+0000 7f7571d9b640 1 -- 172.21.15.123:0/717951677 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7564001c00 con 0x7f75740a98f0 2024-03-25T19:29:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.515+0000 7f7578e31640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.515+0000 7f7578e31640 4 set_mon_vals no callback set 2024-03-25T19:29:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.515+0000 7f7571d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.515+0000 7f7571d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:10.515 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.515+0000 7f7571d9b640 10 monclient: dump: 2024-03-25T19:29:10.515 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:10.516 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:10.516 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:10.516 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:10.516 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:10.516 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:10.516 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:10.516 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:10.516 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:10.516 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:10.517 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.516+0000 7f7578e31640 10 monclient: shutdown 2024-03-25T19:29:10.517 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.516+0000 7f7578e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:10.517 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.516+0000 7f7578e31640 1 -- 172.21.15.123:0/717951677 >> v1:172.21.15.123:6790/0 conn(0x7f75740a98f0 legacy=0x7f75740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.517 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.516+0000 7f7578e31640 1 -- 172.21.15.123:0/717951677 shutdown_connections 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.516+0000 7f7578e31640 1 -- 172.21.15.123:0/717951677 >> 172.21.15.123:0/717951677 conn(0x7f75740b7ea0 msgr2=0x7f75740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.516+0000 7f7578e31640 1 -- 172.21.15.123:0/717951677 shutdown_connections 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.516+0000 7f7578e31640 1 -- 172.21.15.123:0/717951677 wait complete. 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.517+0000 7f7578e31640 10 monclient: build_initial_monmap 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.517+0000 7f7578e31640 10 monclient: monmap: 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:10.518 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:29:10.519 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:29:10.519 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:10.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.517+0000 7f7578e31640 1 librados: starting msgr at 2024-03-25T19:29:10.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.517+0000 7f7578e31640 1 librados: starting objecter 2024-03-25T19:29:10.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.517+0000 7f7578e31640 1 Processor -- start 2024-03-25T19:29:10.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.517+0000 7f7578e31640 1 -- start start 2024-03-25T19:29:10.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.517+0000 7f7578e31640 1 librados: setting wanted keys 2024-03-25T19:29:10.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.517+0000 7f7578e31640 1 librados: calling monclient init 2024-03-25T19:29:10.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.517+0000 7f7578e31640 10 monclient: init 2024-03-25T19:29:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.518+0000 7f7578e31640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.518+0000 7f7578e31640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.518+0000 7f7578e31640 10 monclient(hunting): picked mon.noname-b con 0x7f75740a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.518+0000 7f7578e31640 10 monclient(hunting): picked mon.noname-c con 0x7f757413f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.518+0000 7f7578e31640 10 monclient(hunting): picked mon.noname-a con 0x7f75741430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.518+0000 7f7578e31640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757413be70 con 0x7f75741430c0 2024-03-25T19:29:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.518+0000 7f7578e31640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7574131660 con 0x7f75740a98f0 2024-03-25T19:29:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.518+0000 7f7578e31640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75741317f0 con 0x7f757413f710 2024-03-25T19:29:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.518+0000 7f7578e31640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:10.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.518+0000 7f7578e31640 10 monclient(hunting): authenticate will time out at 5987.878640s 2024-03-25T19:29:10.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.522+0000 7f757259c640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f75740a98f0 0x7f7574130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:51236/0 (socket says 172.21.15.123:51236) 2024-03-25T19:29:10.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.522+0000 7f757259c640 1 -- 172.21.15.123:0/3144869769 learned_addr learned my addr 172.21.15.123:0/3144869769 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.524+0000 7f756affd640 10 client.?.objecter ms_handle_connect 0x7f75740a98f0 2024-03-25T19:29:10.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.524+0000 7f756affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.533+0000 7f756affd640 10 client.?.objecter ms_handle_connect 0x7f757413f710 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.533+0000 7f756affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.535+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208804336 0 0) 0x7f7574131660 con 0x7f75740a98f0 2024-03-25T19:29:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.535+0000 7f756affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:10.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.535+0000 7f756affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:10.536 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.535+0000 7f756affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:10.536 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.535+0000 7f756affd640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:29:10.536 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.535+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7554004eb0 con 0x7f75740a98f0 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.545+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3159441404 0 0) 0x7f7554004eb0 con 0x7f75740a98f0 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.545+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7574131660 con 0x7f75740a98f0 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.545+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7564001690 con 0x7f75740a98f0 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.545+0000 7f756affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.545+0000 7f756affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.545+0000 7f756affd640 10 monclient(hunting): dump: 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:10.545 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:10.546 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:10.546 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:10.546 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:10.546 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:10.546 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:10.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.545+0000 7f756affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:10.550 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.550+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498353309 0 0) 0x7f75741317f0 con 0x7f757413f710 2024-03-25T19:29:10.550 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.550+0000 7f756affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:10.550 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.550+0000 7f756affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:10.550 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.550+0000 7f756affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:10.550 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.550+0000 7f756affd640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:29:10.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.550+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7554004eb0 con 0x7f757413f710 2024-03-25T19:29:10.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.553+0000 7f756affd640 10 client.?.objecter ms_handle_connect 0x7f75741430c0 2024-03-25T19:29:10.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.553+0000 7f756affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:10.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2784681529 0 0) 0x7f7574131660 con 0x7f75740a98f0 2024-03-25T19:29:10.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 >> v1:172.21.15.123:6790/0 conn(0x7f757413f710 legacy=0x7f75741d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 >> v1:172.21.15.123:6789/0 conn(0x7f75741430c0 legacy=0x7f75741e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:10.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 1 monclient: found mon.b 2024-03-25T19:29:10.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:10.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:29:10.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7574131980 con 0x7f75740a98f0 2024-03-25T19:29:10.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 10 monclient: _finish_auth 0 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f756affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f7578e31640 5 monclient: authenticate success, global_id 4516 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f7578e31640 10 monclient: _renew_subs 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f7578e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.557+0000 7f7578e31640 1 -- 172.21.15.123:0/3144869769 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7574135c60 con 0x7f75740a98f0 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.558+0000 7f7578e31640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.558+0000 7f7578e31640 10 monclient: _renew_subs 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.558+0000 7f7578e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.558+0000 7f7578e31640 1 -- 172.21.15.123:0/3144869769 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75740b7770 con 0x7f75740a98f0 2024-03-25T19:29:10.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.558+0000 7f7578e31640 1 librados: init done 2024-03-25T19:29:10.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.559+0000 7f7578e31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:10.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.559+0000 7f7578e31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:10.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.559+0000 7f7578e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:29:10.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.559+0000 7f7578e31640 1 -- 172.21.15.123:0/3144869769 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75741371d0 con 0x7f75740a98f0 2024-03-25T19:29:10.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.831+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75640038a0 con 0x7f75740a98f0 2024-03-25T19:29:10.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.831+0000 7f756affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:10.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.831+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7564003cb0 con 0x7f75740a98f0 2024-03-25T19:29:10.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.831+0000 7f756affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.831+0000 7f756affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.831+0000 7f756affd640 10 monclient: dump: 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:10.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.831+0000 7f757159a640 4 set_mon_vals no callback set 2024-03-25T19:29:10.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.832+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f7564003ed0 con 0x7f75740a98f0 2024-03-25T19:29:10.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.833+0000 7f756bfff640 10 monclient: get_auth_request con 0x7f755406deb0 auth_method 0 2024-03-25T19:29:10.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.833+0000 7f756affd640 10 client.4516.objecter ms_handle_connect 0x7f755406deb0 2024-03-25T19:29:10.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.838+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f7564054980 con 0x7f75740a98f0 2024-03-25T19:29:10.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.838+0000 7f756affd640 10 client.4516.objecter ms_dispatch 0x7f7574000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:10.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.838+0000 7f756affd640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:10.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.838+0000 7f756affd640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:10.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.838+0000 7f756affd640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T19:29:10.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.838+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7564050490 con 0x7f75740a98f0 2024-03-25T19:29:10.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.838+0000 7f756affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:10.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:10.838+0000 7f756affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:11.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.056+0000 7f7578e31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:11.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.056+0000 7f7578e31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:11.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.056+0000 7f7578e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:29:11.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.056+0000 7f7578e31640 1 -- 172.21.15.123:0/3144869769 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75740627b0 con 0x7f75740a98f0 2024-03-25T19:29:11.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.134+0000 7f756affd640 1 -- 172.21.15.123:0/3144869769 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f75640523c0 con 0x7f75740a98f0 2024-03-25T19:29:11.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.134+0000 7f756affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:11.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.134+0000 7f756affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:11.135 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:29:11.135 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:11.137 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 10 librados: watch_flush enter 2024-03-25T19:29:11.137 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 10 librados: watch_flush exit 2024-03-25T19:29:11.137 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T19:29:11.137 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 10 client.4516.objecter successfully canceled tick 2024-03-25T19:29:11.137 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 1 -- 172.21.15.123:0/3144869769 >> v1:172.21.15.151:6800/727684239 conn(0x7f755406deb0 legacy=0x7f7554070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:11.137 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 10 monclient: shutdown 2024-03-25T19:29:11.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:11.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 1 -- 172.21.15.123:0/3144869769 >> v1:172.21.15.151:6789/0 conn(0x7f75740a98f0 legacy=0x7f7574130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:11.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 1 -- 172.21.15.123:0/3144869769 shutdown_connections 2024-03-25T19:29:11.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 1 -- 172.21.15.123:0/3144869769 >> 172.21.15.123:0/3144869769 conn(0x7f75740b7ea0 msgr2=0x7f7574074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:11.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 1 -- 172.21.15.123:0/3144869769 shutdown_connections 2024-03-25T19:29:11.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 1 -- 172.21.15.123:0/3144869769 wait complete. 2024-03-25T19:29:11.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:11.137+0000 7f7578e31640 1 librados: shutdown 2024-03-25T19:29:11.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:15.905 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:07:05.477 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:15.951 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:07:05.484 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:15.957 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:07:05.529 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:15.963 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:07:05.497 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:15.977 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:07:05.549 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:16.008 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:07:05.541 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:16.009 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:07:05.581 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:16.025 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:07:05.597 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:16.050 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:07:05.622 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:17.955 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:18.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.070+0000 7f36b9a63640 10 monclient: get_monmap_and_config 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.070+0000 7f36b9a63640 10 monclient: build_initial_monmap 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.070+0000 7f36b9a63640 10 monclient: monmap: 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:29:18.071462+0000 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:29:18.071462+0000 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:29:18.071 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.071+0000 7f36b9a63640 1 Processor -- start 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.071+0000 7f36b9a63640 1 -- start start 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.071+0000 7f36b9a63640 10 monclient: init 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.072+0000 7f36b9a63640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.072+0000 7f36b9a63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.072+0000 7f36b9a63640 10 monclient(hunting): picked mon.noname-b con 0x7f36b40a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.072+0000 7f36b9a63640 10 monclient(hunting): picked mon.noname-c con 0x7f36b413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.072+0000 7f36b9a63640 10 monclient(hunting): picked mon.noname-a con 0x7f36b41430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.072+0000 7f36b9a63640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b4142520 con 0x7f36b41430c0 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.072+0000 7f36b9a63640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40b28a0 con 0x7f36b413f710 2024-03-25T19:29:18.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.072+0000 7f36b9a63640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40b2a30 con 0x7f36b40a98f0 2024-03-25T19:29:18.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.072+0000 7f36b9a63640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:18.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.072+0000 7f36b9a63640 10 monclient(hunting): authenticate will time out at 5995.432384s 2024-03-25T19:29:18.076 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.076+0000 7f36b37fe640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f36b40a98f0 0x7f36b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:39352/0 (socket says 172.21.15.123:39352) 2024-03-25T19:29:18.076 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.076+0000 7f36b37fe640 1 -- 172.21.15.123:0/1606133927 learned_addr learned my addr 172.21.15.123:0/1606133927 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:18.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.082+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890221056 0 0) 0x7f36b40b2a30 con 0x7f36b40a98f0 2024-03-25T19:29:18.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.082+0000 7f36b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.082+0000 7f36b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.082+0000 7f36b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.082+0000 7f36b27fc640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:29:18.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.082+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f369c003520 con 0x7f36b40a98f0 2024-03-25T19:29:18.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.093+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412449960 0 0) 0x7f36b40b28a0 con 0x7f36b413f710 2024-03-25T19:29:18.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.093+0000 7f36b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.093+0000 7f36b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.093+0000 7f36b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.093+0000 7f36b27fc640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:29:18.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.093+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b40b2a30 con 0x7f36b413f710 2024-03-25T19:29:18.094 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1919617523 0 0) 0x7f369c003520 con 0x7f36b40a98f0 2024-03-25T19:29:18.094 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 >> v1:172.21.15.151:6789/0 conn(0x7f36b413f710 legacy=0x7f36b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 >> v1:172.21.15.123:6789/0 conn(0x7f36b41430c0 legacy=0x7f36b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 1 monclient: found mon.noname-b 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b40b44c0 con 0x7f36b40a98f0 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 10 monclient: _finish_auth 0 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f36a4001690 con 0x7f36b40a98f0 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b27fc640 10 monclient: dump: 2024-03-25T19:29:18.095 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b9a63640 5 monclient: authenticate success, global_id 4541 2024-03-25T19:29:18.096 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.094+0000 7f36b9a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:18.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.098+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a40017f0 con 0x7f36b40a98f0 2024-03-25T19:29:18.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.098+0000 7f36b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:18.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.098+0000 7f36b27fc640 1 -- 172.21.15.123:0/1606133927 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f36a4001c00 con 0x7f36b40a98f0 2024-03-25T19:29:18.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.098+0000 7f36b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:18.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.098+0000 7f36b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:18.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.098+0000 7f36b27fc640 10 monclient: dump: 2024-03-25T19:29:18.098 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:18.098 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:18.098 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:18.098 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:18.099 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:18.099 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:18.099 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:18.099 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:18.099 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:18.099 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:18.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.098+0000 7f36b9a63640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:18.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.098+0000 7f36b9a63640 4 set_mon_vals no callback set 2024-03-25T19:29:18.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.099+0000 7f36b9a63640 10 monclient: shutdown 2024-03-25T19:29:18.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.099+0000 7f36b9a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:18.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.099+0000 7f36b9a63640 1 -- 172.21.15.123:0/1606133927 >> v1:172.21.15.123:6790/0 conn(0x7f36b40a98f0 legacy=0x7f36b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.099+0000 7f36b9a63640 1 -- 172.21.15.123:0/1606133927 shutdown_connections 2024-03-25T19:29:18.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.099+0000 7f36b9a63640 1 -- 172.21.15.123:0/1606133927 >> 172.21.15.123:0/1606133927 conn(0x7f36b40b7ea0 msgr2=0x7f36b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.099+0000 7f36b9a63640 1 -- 172.21.15.123:0/1606133927 shutdown_connections 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.099+0000 7f36b9a63640 1 -- 172.21.15.123:0/1606133927 wait complete. 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.100+0000 7f36b9a63640 10 monclient: build_initial_monmap 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.100+0000 7f36b9a63640 10 monclient: monmap: 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:18.101 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.100+0000 7f36b9a63640 1 librados: starting msgr at 2024-03-25T19:29:18.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.100+0000 7f36b9a63640 1 librados: starting objecter 2024-03-25T19:29:18.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.100+0000 7f36b9a63640 1 Processor -- start 2024-03-25T19:29:18.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.101+0000 7f36b9a63640 1 -- start start 2024-03-25T19:29:18.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.101+0000 7f36b9a63640 1 librados: setting wanted keys 2024-03-25T19:29:18.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.101+0000 7f36b9a63640 1 librados: calling monclient init 2024-03-25T19:29:18.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.101+0000 7f36b9a63640 10 monclient: init 2024-03-25T19:29:18.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.101+0000 7f36b9a63640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:18.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.101+0000 7f36b9a63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:18.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.101+0000 7f36b9a63640 10 monclient(hunting): picked mon.noname-c con 0x7f36b40a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:18.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.101+0000 7f36b9a63640 10 monclient(hunting): picked mon.noname-a con 0x7f36b413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:18.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.102+0000 7f36b9a63640 10 monclient(hunting): picked mon.noname-b con 0x7f36b41430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:18.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.102+0000 7f36b9a63640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b4138b20 con 0x7f36b413f710 2024-03-25T19:29:18.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.102+0000 7f36b9a63640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b4137a80 con 0x7f36b41430c0 2024-03-25T19:29:18.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.102+0000 7f36b9a63640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b4139880 con 0x7f36b40a98f0 2024-03-25T19:29:18.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.102+0000 7f36b9a63640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:18.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.102+0000 7f36b9a63640 10 monclient(hunting): authenticate will time out at 5995.461958s 2024-03-25T19:29:18.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.114+0000 7f36b2ffd640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f36b413f710 0x7f36b4139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:39622/0 (socket says 172.21.15.123:39622) 2024-03-25T19:29:18.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.114+0000 7f36b2ffd640 1 -- 172.21.15.123:0/3385163932 learned_addr learned my addr 172.21.15.123:0/3385163932 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:18.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.115+0000 7f369bfff640 10 client.?.objecter ms_handle_connect 0x7f36b413f710 2024-03-25T19:29:18.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.115+0000 7f369bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:18.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.124+0000 7f369bfff640 10 client.?.objecter ms_handle_connect 0x7f36b41430c0 2024-03-25T19:29:18.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.124+0000 7f369bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:18.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.125+0000 7f369bfff640 10 client.?.objecter ms_handle_connect 0x7f36b40a98f0 2024-03-25T19:29:18.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.125+0000 7f369bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:18.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725083750 0 0) 0x7f36b4139880 con 0x7f36b40a98f0 2024-03-25T19:29:18.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:29:18.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a0004eb0 con 0x7f36b40a98f0 2024-03-25T19:29:18.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862953371 0 0) 0x7f36b4138b20 con 0x7f36b413f710 2024-03-25T19:29:18.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:29:18.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.130+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b4139880 con 0x7f36b413f710 2024-03-25T19:29:18.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.141+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9469340 0 0) 0x7f36b4137a80 con 0x7f36b41430c0 2024-03-25T19:29:18.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.141+0000 7f369bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.141+0000 7f369bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.141+0000 7f369bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.141+0000 7f369bfff640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:29:18.142 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.141+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b4138b20 con 0x7f36b41430c0 2024-03-25T19:29:18.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.144+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3738017073 0 0) 0x7f36b4139880 con 0x7f36b413f710 2024-03-25T19:29:18.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.144+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36b4137a80 con 0x7f36b413f710 2024-03-25T19:29:18.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.144+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f36a8001a70 con 0x7f36b413f710 2024-03-25T19:29:18.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.144+0000 7f369bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:18.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.144+0000 7f369bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:18.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.144+0000 7f369bfff640 10 monclient(hunting): dump: 2024-03-25T19:29:18.144 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:18.145 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:18.145 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:18.145 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:18.145 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:18.145 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:18.145 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:18.145 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:18.145 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:18.145 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:18.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.144+0000 7f369bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:18.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.147+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 895723084 0 0) 0x7f36a0004eb0 con 0x7f36b40a98f0 2024-03-25T19:29:18.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.147+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36b4139880 con 0x7f36b40a98f0 2024-03-25T19:29:18.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.147+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f36a4001b00 con 0x7f36b40a98f0 2024-03-25T19:29:18.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.147+0000 7f369bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:18.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.147+0000 7f369bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.147+0000 7f369bfff640 10 monclient(hunting): dump: 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:18.148 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1074780759 0 0) 0x7f36b4139880 con 0x7f36b40a98f0 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 >> v1:172.21.15.151:6789/0 conn(0x7f36b41430c0 legacy=0x7f36b41e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 >> v1:172.21.15.123:6789/0 conn(0x7f36b413f710 legacy=0x7f36b4139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 1 monclient: found mon.c 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4139a10 con 0x7f36b40a98f0 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 10 monclient: _finish_auth 0 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:18.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.153+0000 7f369bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:18.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.154+0000 7f36b9a63640 5 monclient: authenticate success, global_id 4544 2024-03-25T19:29:18.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.154+0000 7f36b9a63640 10 monclient: _renew_subs 2024-03-25T19:29:18.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.154+0000 7f36b9a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:18.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.154+0000 7f36b9a63640 1 -- 172.21.15.123:0/3385163932 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b41dab00 con 0x7f36b40a98f0 2024-03-25T19:29:18.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.154+0000 7f36b9a63640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:18.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.154+0000 7f36b9a63640 10 monclient: _renew_subs 2024-03-25T19:29:18.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.154+0000 7f36b9a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:18.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.154+0000 7f36b9a63640 1 -- 172.21.15.123:0/3385163932 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b413b4f0 con 0x7f36b40a98f0 2024-03-25T19:29:18.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.154+0000 7f36b9a63640 1 librados: init done 2024-03-25T19:29:18.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.155+0000 7f36b9a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:18.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.155+0000 7f36b9a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:18.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.155+0000 7f36b9a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:18.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.155+0000 7f36b9a63640 1 -- 172.21.15.123:0/3385163932 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b41dae00 con 0x7f36b40a98f0 2024-03-25T19:29:18.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.446+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a4001220 con 0x7f36b40a98f0 2024-03-25T19:29:18.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.446+0000 7f369bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:18.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.446+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f36a4004c30 con 0x7f36b40a98f0 2024-03-25T19:29:18.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.446+0000 7f369bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.446+0000 7f369bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.446+0000 7f369bfff640 10 monclient: dump: 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:18.447 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.446+0000 7f36b1ffb640 4 set_mon_vals no callback set 2024-03-25T19:29:18.448 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.447+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f36a401ac40 con 0x7f36b40a98f0 2024-03-25T19:29:18.448 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.447+0000 7f36b2ffd640 10 monclient: get_auth_request con 0x7f36a006dfb0 auth_method 0 2024-03-25T19:29:18.448 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.448+0000 7f369bfff640 10 client.4544.objecter ms_handle_connect 0x7f36a006dfb0 2024-03-25T19:29:18.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.454+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f36a4054840 con 0x7f36b40a98f0 2024-03-25T19:29:18.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.454+0000 7f369bfff640 10 client.4544.objecter ms_dispatch 0x7f36b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:18.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.454+0000 7f369bfff640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:18.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.454+0000 7f369bfff640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:18.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.454+0000 7f369bfff640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-25T19:29:18.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.454+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36a4050180 con 0x7f36b40a98f0 2024-03-25T19:29:18.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.454+0000 7f369bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:18.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.454+0000 7f369bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:18.689 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.689+0000 7f36b9a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:18.690 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.689+0000 7f36b9a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:18.690 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.689+0000 7f36b9a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:18.690 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.689+0000 7f36b9a63640 1 -- 172.21.15.123:0/3385163932 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36b40b3e80 con 0x7f36b40a98f0 2024-03-25T19:29:18.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.781+0000 7f369bfff640 1 -- 172.21.15.123:0/3385163932 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f36a4052280 con 0x7f36b40a98f0 2024-03-25T19:29:18.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.781+0000 7f369bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:18.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.781+0000 7f369bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:18.782 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:29:18.782 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:18.784 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.783+0000 7f36b9a63640 10 librados: watch_flush enter 2024-03-25T19:29:18.784 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.783+0000 7f36b9a63640 10 librados: watch_flush exit 2024-03-25T19:29:18.784 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.783+0000 7f36b9a63640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-25T19:29:18.784 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.783+0000 7f36b9a63640 10 client.4544.objecter successfully canceled tick 2024-03-25T19:29:18.784 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.783+0000 7f36b9a63640 1 -- 172.21.15.123:0/3385163932 >> v1:172.21.15.151:6800/727684239 conn(0x7f36a006dfb0 legacy=0x7f36a0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.785 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.783+0000 7f36b9a63640 10 monclient: shutdown 2024-03-25T19:29:18.785 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.783+0000 7f36b9a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:18.785 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.783+0000 7f36b9a63640 1 -- 172.21.15.123:0/3385163932 >> v1:172.21.15.123:6790/0 conn(0x7f36b40a98f0 legacy=0x7f36b4137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.785 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.783+0000 7f36b9a63640 1 -- 172.21.15.123:0/3385163932 shutdown_connections 2024-03-25T19:29:18.785 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.783+0000 7f36b9a63640 1 -- 172.21.15.123:0/3385163932 >> 172.21.15.123:0/3385163932 conn(0x7f36b40b7ea0 msgr2=0x7f36b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:18.785 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.784+0000 7f36b9a63640 1 -- 172.21.15.123:0/3385163932 shutdown_connections 2024-03-25T19:29:18.785 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.784+0000 7f36b9a63640 1 -- 172.21.15.123:0/3385163932 wait complete. 2024-03-25T19:29:18.785 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:18.784+0000 7f36b9a63640 1 librados: shutdown 2024-03-25T19:29:18.795 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:25.602 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:25.773 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.773+0000 7f7a71ec1640 10 monclient: get_monmap_and_config 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.773+0000 7f7a71ec1640 10 monclient: build_initial_monmap 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.773+0000 7f7a71ec1640 10 monclient: monmap: 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:29:25.774713+0000 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:29:25.774713+0000 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:29:25.774 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.773+0000 7f7a71ec1640 1 Processor -- start 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.773+0000 7f7a71ec1640 1 -- start start 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.773+0000 7f7a71ec1640 10 monclient: init 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.774+0000 7f7a71ec1640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.774+0000 7f7a71ec1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.774+0000 7f7a71ec1640 10 monclient(hunting): picked mon.noname-c con 0x7f7a6c0a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.774+0000 7f7a71ec1640 10 monclient(hunting): picked mon.noname-a con 0x7f7a6c13f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.774+0000 7f7a71ec1640 10 monclient(hunting): picked mon.noname-b con 0x7f7a6c1430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.774+0000 7f7a71ec1640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c142520 con 0x7f7a6c13f710 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.774+0000 7f7a71ec1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c0b28a0 con 0x7f7a6c0a98f0 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.774+0000 7f7a71ec1640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c0b2a30 con 0x7f7a6c1430c0 2024-03-25T19:29:25.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.774+0000 7f7a71ec1640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:25.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.774+0000 7f7a71ec1640 10 monclient(hunting): authenticate will time out at 6003.135239s 2024-03-25T19:29:25.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.775+0000 7f7a6bfff640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f7a6c1430c0 0x7f7a6c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:52850/0 (socket says 172.21.15.123:52850) 2024-03-25T19:29:25.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.775+0000 7f7a6bfff640 1 -- 172.21.15.123:0/3100164034 learned_addr learned my addr 172.21.15.123:0/3100164034 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:25.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.789+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347370299 0 0) 0x7f7a6c0b2a30 con 0x7f7a6c1430c0 2024-03-25T19:29:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.789+0000 7f7a6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.789+0000 7f7a6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.789+0000 7f7a6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.789+0000 7f7a6a7fc640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:29:25.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.789+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a54003520 con 0x7f7a6c1430c0 2024-03-25T19:29:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.801+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735124341 0 0) 0x7f7a6c0b28a0 con 0x7f7a6c0a98f0 2024-03-25T19:29:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.801+0000 7f7a6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.801+0000 7f7a6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.801+0000 7f7a6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.801+0000 7f7a6a7fc640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:29:25.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.801+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6c0b2a30 con 0x7f7a6c0a98f0 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.808+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285461796 0 0) 0x7f7a54003520 con 0x7f7a6c1430c0 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.808+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 >> v1:172.21.15.151:6789/0 conn(0x7f7a6c0a98f0 legacy=0x7f7a6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.808+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 >> v1:172.21.15.123:6789/0 conn(0x7f7a6c13f710 legacy=0x7f7a6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.808+0000 7f7a6a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.808+0000 7f7a6a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.808+0000 7f7a6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.808+0000 7f7a6a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.808+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c0b44c0 con 0x7f7a6c1430c0 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.809+0000 7f7a6a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.809+0000 7f7a6a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:29:25.810 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.809+0000 7f7a6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.809+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7a64001a10 con 0x7f7a6c1430c0 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.809+0000 7f7a6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.809+0000 7f7a6a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.809+0000 7f7a6a7fc640 10 monclient: dump: 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:25.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.809+0000 7f7a71ec1640 5 monclient: authenticate success, global_id 4547 2024-03-25T19:29:25.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.809+0000 7f7a71ec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:25.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.813+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a64001b70 con 0x7f7a6c1430c0 2024-03-25T19:29:25.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.813+0000 7f7a6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:25.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.813+0000 7f7a6a7fc640 1 -- 172.21.15.123:0/3100164034 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7a64001f80 con 0x7f7a6c1430c0 2024-03-25T19:29:25.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.813+0000 7f7a6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:25.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.813+0000 7f7a6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:25.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.813+0000 7f7a6a7fc640 10 monclient: dump: 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.813+0000 7f7a71ec1640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:25.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.813+0000 7f7a71ec1640 4 set_mon_vals no callback set 2024-03-25T19:29:25.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.814+0000 7f7a71ec1640 10 monclient: shutdown 2024-03-25T19:29:25.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.814+0000 7f7a71ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:25.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.814+0000 7f7a71ec1640 1 -- 172.21.15.123:0/3100164034 >> v1:172.21.15.123:6790/0 conn(0x7f7a6c1430c0 legacy=0x7f7a6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.814+0000 7f7a71ec1640 1 -- 172.21.15.123:0/3100164034 shutdown_connections 2024-03-25T19:29:25.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.814+0000 7f7a71ec1640 1 -- 172.21.15.123:0/3100164034 >> 172.21.15.123:0/3100164034 conn(0x7f7a6c0b7ea0 msgr2=0x7f7a6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:25.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.814+0000 7f7a71ec1640 1 -- 172.21.15.123:0/3100164034 shutdown_connections 2024-03-25T19:29:25.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.814+0000 7f7a71ec1640 1 -- 172.21.15.123:0/3100164034 wait complete. 2024-03-25T19:29:25.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.815+0000 7f7a71ec1640 10 monclient: build_initial_monmap 2024-03-25T19:29:25.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.815+0000 7f7a71ec1640 10 monclient: monmap: 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.815+0000 7f7a71ec1640 1 librados: starting msgr at 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.815+0000 7f7a71ec1640 1 librados: starting objecter 2024-03-25T19:29:25.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.815+0000 7f7a71ec1640 1 Processor -- start 2024-03-25T19:29:25.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.816+0000 7f7a71ec1640 1 -- start start 2024-03-25T19:29:25.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.816+0000 7f7a71ec1640 1 librados: setting wanted keys 2024-03-25T19:29:25.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.816+0000 7f7a71ec1640 1 librados: calling monclient init 2024-03-25T19:29:25.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.816+0000 7f7a71ec1640 10 monclient: init 2024-03-25T19:29:25.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.816+0000 7f7a71ec1640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:25.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.816+0000 7f7a71ec1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:25.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.817+0000 7f7a71ec1640 10 monclient(hunting): picked mon.noname-b con 0x7f7a6c0a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:25.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.817+0000 7f7a71ec1640 10 monclient(hunting): picked mon.noname-a con 0x7f7a6c13f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:25.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.817+0000 7f7a71ec1640 10 monclient(hunting): picked mon.noname-c con 0x7f7a6c1430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:25.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.817+0000 7f7a71ec1640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c137ab0 con 0x7f7a6c13f710 2024-03-25T19:29:25.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.817+0000 7f7a71ec1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c138bd0 con 0x7f7a6c0a98f0 2024-03-25T19:29:25.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.817+0000 7f7a71ec1640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c138e40 con 0x7f7a6c1430c0 2024-03-25T19:29:25.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.817+0000 7f7a71ec1640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:25.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.817+0000 7f7a71ec1640 10 monclient(hunting): authenticate will time out at 6003.178505s 2024-03-25T19:29:25.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.821+0000 7f7a6bfff640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f7a6c1430c0 0x7f7a6c1e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:52874/0 (socket says 172.21.15.123:52874) 2024-03-25T19:29:25.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.821+0000 7f7a6bfff640 1 -- 172.21.15.123:0/1728995333 learned_addr learned my addr 172.21.15.123:0/1728995333 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:25.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.822+0000 7f7a53fff640 10 client.?.objecter ms_handle_connect 0x7f7a6c1430c0 2024-03-25T19:29:25.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.822+0000 7f7a53fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:25.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.826+0000 7f7a53fff640 10 client.?.objecter ms_handle_connect 0x7f7a6c0a98f0 2024-03-25T19:29:25.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.826+0000 7f7a53fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:25.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.827+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705459635 0 0) 0x7f7a6c138e40 con 0x7f7a6c1430c0 2024-03-25T19:29:25.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.827+0000 7f7a53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:25.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.827+0000 7f7a53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:25.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.827+0000 7f7a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:25.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.827+0000 7f7a53fff640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:29:25.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.827+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a58004eb0 con 0x7f7a6c1430c0 2024-03-25T19:29:25.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.830+0000 7f7a53fff640 10 client.?.objecter ms_handle_connect 0x7f7a6c13f710 2024-03-25T19:29:25.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.830+0000 7f7a53fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:25.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.834+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120980132 0 0) 0x7f7a6c138bd0 con 0x7f7a6c0a98f0 2024-03-25T19:29:25.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.834+0000 7f7a53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:25.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.834+0000 7f7a53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:25.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.834+0000 7f7a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:25.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.834+0000 7f7a53fff640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:29:25.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.834+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6c138e40 con 0x7f7a6c0a98f0 2024-03-25T19:29:25.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.843+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1889905942 0 0) 0x7f7a58004eb0 con 0x7f7a6c1430c0 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.843+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a6c138bd0 con 0x7f7a6c1430c0 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.843+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7a64001ec0 con 0x7f7a6c1430c0 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.843+0000 7f7a53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.843+0000 7f7a53fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.843+0000 7f7a53fff640 10 monclient(hunting): dump: 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:25.845 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:25.846 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:25.846 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:25.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.843+0000 7f7a53fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:25.848 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.848+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2580224657 0 0) 0x7f7a6c138e40 con 0x7f7a6c0a98f0 2024-03-25T19:29:25.848 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.848+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a58004eb0 con 0x7f7a6c0a98f0 2024-03-25T19:29:25.848 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.848+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7a5c001930 con 0x7f7a6c0a98f0 2024-03-25T19:29:25.848 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.848+0000 7f7a53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:25.848 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.848+0000 7f7a53fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.848+0000 7f7a53fff640 10 monclient(hunting): dump: 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:25.849 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:25.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1540222275 0 0) 0x7f7a6c138bd0 con 0x7f7a6c1430c0 2024-03-25T19:29:25.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 >> v1:172.21.15.151:6789/0 conn(0x7f7a6c0a98f0 legacy=0x7f7a6c137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 >> v1:172.21.15.123:6789/0 conn(0x7f7a6c13f710 legacy=0x7f7a6c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:25.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 1 monclient: found mon.c 2024-03-25T19:29:25.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:25.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:25.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c139120 con 0x7f7a6c1430c0 2024-03-25T19:29:25.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 10 monclient: _finish_auth 0 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a71ec1640 5 monclient: authenticate success, global_id 4550 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a71ec1640 10 monclient: _renew_subs 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a71ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a71ec1640 1 -- 172.21.15.123:0/1728995333 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c139400 con 0x7f7a6c1430c0 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a71ec1640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a71ec1640 10 monclient: _renew_subs 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a71ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a71ec1640 1 -- 172.21.15.123:0/1728995333 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c0b7880 con 0x7f7a6c1430c0 2024-03-25T19:29:25.854 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.853+0000 7f7a71ec1640 1 librados: init done 2024-03-25T19:29:25.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.854+0000 7f7a71ec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:25.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.854+0000 7f7a71ec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:25.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.854+0000 7f7a71ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:25.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:25.854+0000 7f7a71ec1640 1 -- 172.21.15.123:0/1728995333 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c137810 con 0x7f7a6c1430c0 2024-03-25T19:29:26.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.153+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a64004e10 con 0x7f7a6c1430c0 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.153+0000 7f7a53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.153+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7a64002100 con 0x7f7a6c1430c0 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.153+0000 7f7a53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.153+0000 7f7a53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.153+0000 7f7a53fff640 10 monclient: dump: 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:26.154 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:26.155 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:26.155 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:26.155 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:26.155 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:26.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.153+0000 7f7a69ffb640 4 set_mon_vals no callback set 2024-03-25T19:29:26.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.154+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f7a6401af90 con 0x7f7a6c1430c0 2024-03-25T19:29:26.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.155+0000 7f7a6b7fe640 10 monclient: get_auth_request con 0x7f7a5806de40 auth_method 0 2024-03-25T19:29:26.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.155+0000 7f7a53fff640 10 client.4550.objecter ms_handle_connect 0x7f7a5806de40 2024-03-25T19:29:26.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.160+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f7a64054b80 con 0x7f7a6c1430c0 2024-03-25T19:29:26.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.160+0000 7f7a53fff640 10 client.4550.objecter ms_dispatch 0x7f7a6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:26.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.160+0000 7f7a53fff640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:26.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.160+0000 7f7a53fff640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:26.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.160+0000 7f7a53fff640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-25T19:29:26.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.160+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a640504c0 con 0x7f7a6c1430c0 2024-03-25T19:29:26.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.160+0000 7f7a53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:26.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.160+0000 7f7a53fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:26.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.404+0000 7f7a71ec1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:26.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.405+0000 7f7a71ec1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:26.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.405+0000 7f7a71ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:26.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.405+0000 7f7a71ec1640 1 -- 172.21.15.123:0/1728995333 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a6c137c60 con 0x7f7a6c1430c0 2024-03-25T19:29:26.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.427+0000 7f7a53fff640 1 -- 172.21.15.123:0/1728995333 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f7a6c137c60 con 0x7f7a6c1430c0 2024-03-25T19:29:26.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.427+0000 7f7a53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:26.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.427+0000 7f7a53fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:26.427 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:29:26.427 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 10 librados: watch_flush enter 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 10 librados: watch_flush exit 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 10 client.4550.objecter successfully canceled tick 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 1 -- 172.21.15.123:0/1728995333 >> v1:172.21.15.151:6800/727684239 conn(0x7f7a5806de40 legacy=0x7f7a58070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 10 monclient: shutdown 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 1 -- 172.21.15.123:0/1728995333 >> v1:172.21.15.123:6790/0 conn(0x7f7a6c1430c0 legacy=0x7f7a6c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 1 -- 172.21.15.123:0/1728995333 shutdown_connections 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 1 -- 172.21.15.123:0/1728995333 >> 172.21.15.123:0/1728995333 conn(0x7f7a6c0b7ea0 msgr2=0x7f7a6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 1 -- 172.21.15.123:0/1728995333 shutdown_connections 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 1 -- 172.21.15.123:0/1728995333 wait complete. 2024-03-25T19:29:26.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:26.429+0000 7f7a71ec1640 1 librados: shutdown 2024-03-25T19:29:26.441 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:33.248 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:33.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.365+0000 7f197f3cd640 10 monclient: get_monmap_and_config 2024-03-25T19:29:33.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.365+0000 7f197f3cd640 10 monclient: build_initial_monmap 2024-03-25T19:29:33.365 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.365+0000 7f197f3cd640 10 monclient: monmap: 2024-03-25T19:29:33.366 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:33.366 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:33.366 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:29:33.366539+0000 2024-03-25T19:29:33.366 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:29:33.366539+0000 2024-03-25T19:29:33.366 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:33.366 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:33.366 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:33.366 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:29:33.366 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:29:33.366 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.366+0000 7f197f3cd640 1 Processor -- start 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.366+0000 7f197f3cd640 1 -- start start 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.366+0000 7f197f3cd640 10 monclient: init 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.367+0000 7f197f3cd640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.367+0000 7f197f3cd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.367+0000 7f197f3cd640 10 monclient(hunting): picked mon.noname-a con 0x7f19780a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.367+0000 7f197f3cd640 10 monclient(hunting): picked mon.noname-c con 0x7f197813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.367+0000 7f197f3cd640 10 monclient(hunting): picked mon.noname-b con 0x7f19781430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.367+0000 7f197f3cd640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1978142520 con 0x7f19780a98f0 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.367+0000 7f197f3cd640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19780b28a0 con 0x7f197813f710 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.367+0000 7f197f3cd640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19780b2a30 con 0x7f19781430c0 2024-03-25T19:29:33.368 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.367+0000 7f197f3cd640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:33.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.367+0000 7f197f3cd640 10 monclient(hunting): authenticate will time out at 6010.727186s 2024-03-25T19:29:33.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.371+0000 7f197c968640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f197813f710 0x7f19781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:35216/0 (socket says 172.21.15.123:35216) 2024-03-25T19:29:33.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.371+0000 7f197c968640 1 -- 172.21.15.123:0/1729049807 learned_addr learned my addr 172.21.15.123:0/1729049807 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:33.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.386+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143868529 0 0) 0x7f19780b28a0 con 0x7f197813f710 2024-03-25T19:29:33.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.386+0000 7f196ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.386+0000 7f196ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.386+0000 7f196ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.386+0000 7f196ffff640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:29:33.386 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.386+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1964003520 con 0x7f197813f710 2024-03-25T19:29:33.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.388+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766086810 0 0) 0x7f1978142520 con 0x7f19780a98f0 2024-03-25T19:29:33.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.388+0000 7f196ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.388+0000 7f196ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.388+0000 7f196ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.388+0000 7f196ffff640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:29:33.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.388+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19780b28a0 con 0x7f19780a98f0 2024-03-25T19:29:33.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.400+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306684565 0 0) 0x7f19780b2a30 con 0x7f19781430c0 2024-03-25T19:29:33.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.400+0000 7f196ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.400+0000 7f196ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.400+0000 7f196ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.400+0000 7f196ffff640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:29:33.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.400+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1978142520 con 0x7f19781430c0 2024-03-25T19:29:33.401 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 825874698 0 0) 0x7f19780b28a0 con 0x7f19780a98f0 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 >> v1:172.21.15.123:6790/0 conn(0x7f19781430c0 legacy=0x7f197814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 >> v1:172.21.15.151:6789/0 conn(0x7f197813f710 legacy=0x7f19781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 1 monclient: found mon.noname-a 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19780b44c0 con 0x7f19780a98f0 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 10 monclient: _finish_auth 0 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1968001670 con 0x7f19780a98f0 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:33.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f196ffff640 10 monclient: dump: 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f197f3cd640 5 monclient: authenticate success, global_id 4521 2024-03-25T19:29:33.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.401+0000 7f197f3cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:33.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.412+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19680017d0 con 0x7f19780a98f0 2024-03-25T19:29:33.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.412+0000 7f196ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:33.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.412+0000 7f196ffff640 1 -- 172.21.15.123:0/1729049807 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1968001be0 con 0x7f19780a98f0 2024-03-25T19:29:33.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.412+0000 7f196ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.412+0000 7f196ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.412+0000 7f196ffff640 10 monclient: dump: 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.412+0000 7f197f3cd640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:33.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.412+0000 7f197f3cd640 4 set_mon_vals no callback set 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.413+0000 7f197f3cd640 10 monclient: shutdown 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.413+0000 7f197f3cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.413+0000 7f197f3cd640 1 -- 172.21.15.123:0/1729049807 >> v1:172.21.15.123:6789/0 conn(0x7f19780a98f0 legacy=0x7f19780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.413+0000 7f197f3cd640 1 -- 172.21.15.123:0/1729049807 shutdown_connections 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.413+0000 7f197f3cd640 1 -- 172.21.15.123:0/1729049807 >> 172.21.15.123:0/1729049807 conn(0x7f19780b7ea0 msgr2=0x7f19780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.413+0000 7f197f3cd640 1 -- 172.21.15.123:0/1729049807 shutdown_connections 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.413+0000 7f197f3cd640 1 -- 172.21.15.123:0/1729049807 wait complete. 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.414+0000 7f197f3cd640 10 monclient: build_initial_monmap 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.414+0000 7f197f3cd640 10 monclient: monmap: 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:33.415 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.414+0000 7f197f3cd640 1 librados: starting msgr at 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.414+0000 7f197f3cd640 1 librados: starting objecter 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.415+0000 7f197f3cd640 1 Processor -- start 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.415+0000 7f197f3cd640 1 -- start start 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.415+0000 7f197f3cd640 1 librados: setting wanted keys 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.415+0000 7f197f3cd640 1 librados: calling monclient init 2024-03-25T19:29:33.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.415+0000 7f197f3cd640 10 monclient: init 2024-03-25T19:29:33.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.415+0000 7f197f3cd640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:33.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.415+0000 7f197f3cd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:33.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.416+0000 7f197f3cd640 10 monclient(hunting): picked mon.noname-c con 0x7f19780a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:33.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.416+0000 7f197f3cd640 10 monclient(hunting): picked mon.noname-a con 0x7f197813f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:33.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.416+0000 7f197f3cd640 10 monclient(hunting): picked mon.noname-b con 0x7f19781430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:33.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.416+0000 7f197f3cd640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1978138f10 con 0x7f197813f710 2024-03-25T19:29:33.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.416+0000 7f197f3cd640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1978139970 con 0x7f19781430c0 2024-03-25T19:29:33.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.416+0000 7f197f3cd640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19780abb50 con 0x7f19780a98f0 2024-03-25T19:29:33.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.416+0000 7f197f3cd640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:33.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.416+0000 7f197f3cd640 10 monclient(hunting): authenticate will time out at 6010.775947s 2024-03-25T19:29:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.420+0000 7f197d169640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f19780a98f0 0x7f1978136330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:57860/0 (socket says 172.21.15.123:57860) 2024-03-25T19:29:33.420 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.420+0000 7f197d169640 1 -- 172.21.15.123:0/3718300093 learned_addr learned my addr 172.21.15.123:0/3718300093 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:33.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.421+0000 7f196dffb640 10 client.?.objecter ms_handle_connect 0x7f19780a98f0 2024-03-25T19:29:33.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.421+0000 7f196dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:33.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.429+0000 7f196dffb640 10 client.?.objecter ms_handle_connect 0x7f197813f710 2024-03-25T19:29:33.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.429+0000 7f196dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:33.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.429+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224581952 0 0) 0x7f19780abb50 con 0x7f19780a98f0 2024-03-25T19:29:33.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.429+0000 7f196dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.429+0000 7f196dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.429+0000 7f196dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.429+0000 7f196dffb640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:29:33.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.429+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f195c004eb0 con 0x7f19780a98f0 2024-03-25T19:29:33.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.435+0000 7f196dffb640 10 client.?.objecter ms_handle_connect 0x7f19781430c0 2024-03-25T19:29:33.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.435+0000 7f196dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:33.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.454+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46943107 0 0) 0x7f1978138f10 con 0x7f197813f710 2024-03-25T19:29:33.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.454+0000 7f196dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.454+0000 7f196dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.454+0000 7f196dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.454+0000 7f196dffb640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:29:33.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.454+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19780abb50 con 0x7f197813f710 2024-03-25T19:29:33.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.460+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3397896646 0 0) 0x7f195c004eb0 con 0x7f19780a98f0 2024-03-25T19:29:33.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.460+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1978138f10 con 0x7f19780a98f0 2024-03-25T19:29:33.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.460+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1968001670 con 0x7f19780a98f0 2024-03-25T19:29:33.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.460+0000 7f196dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.460+0000 7f196dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:33.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.460+0000 7f196dffb640 10 monclient(hunting): dump: 2024-03-25T19:29:33.460 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:33.460 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:33.461 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:33.461 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:33.461 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.461 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:33.461 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:33.461 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:33.461 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:33.461 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:33.461 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.460+0000 7f196dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:33.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.462+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899042519 0 0) 0x7f1978139970 con 0x7f19781430c0 2024-03-25T19:29:33.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.462+0000 7f196dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.462+0000 7f196dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.462+0000 7f196dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.462+0000 7f196dffb640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:29:33.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.462+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f195c004eb0 con 0x7f19781430c0 2024-03-25T19:29:33.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2610714033 0 0) 0x7f1978138f10 con 0x7f19780a98f0 2024-03-25T19:29:33.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 >> v1:172.21.15.151:6789/0 conn(0x7f19781430c0 legacy=0x7f1978138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 >> v1:172.21.15.123:6789/0 conn(0x7f197813f710 legacy=0x7f19780ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:29:33.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 1 monclient: found mon.c 2024-03-25T19:29:33.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:33.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:33.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19780abe30 con 0x7f19780a98f0 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 10 monclient: _finish_auth 0 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f196dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f197f3cd640 5 monclient: authenticate success, global_id 4556 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f197f3cd640 10 monclient: _renew_subs 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f197f3cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f197f3cd640 1 -- 172.21.15.123:0/3718300093 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1978137b40 con 0x7f19780a98f0 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f197f3cd640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f197f3cd640 10 monclient: _renew_subs 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f197f3cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f197f3cd640 1 -- 172.21.15.123:0/3718300093 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1978138c10 con 0x7f19780a98f0 2024-03-25T19:29:33.470 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.469+0000 7f197f3cd640 1 librados: init done 2024-03-25T19:29:33.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.470+0000 7f197f3cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:33.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.470+0000 7f197f3cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:33.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.470+0000 7f197f3cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:33.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.470+0000 7f197f3cd640 1 -- 172.21.15.123:0/3718300093 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19780627b0 con 0x7f19780a98f0 2024-03-25T19:29:33.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.764+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19680038a0 con 0x7f19780a98f0 2024-03-25T19:29:33.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.764+0000 7f196dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:33.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.764+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1968003cb0 con 0x7f19780a98f0 2024-03-25T19:29:33.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.764+0000 7f196dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.764+0000 7f196dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.764+0000 7f196dffb640 10 monclient: dump: 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:33.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.764+0000 7f196f7fe640 4 set_mon_vals no callback set 2024-03-25T19:29:33.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.765+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f1968003ed0 con 0x7f19780a98f0 2024-03-25T19:29:33.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.766+0000 7f197c968640 10 monclient: get_auth_request con 0x7f195c06de00 auth_method 0 2024-03-25T19:29:33.766 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.766+0000 7f196dffb640 10 client.4556.objecter ms_handle_connect 0x7f195c06de00 2024-03-25T19:29:33.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.771+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f1968054930 con 0x7f19780a98f0 2024-03-25T19:29:33.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.771+0000 7f196dffb640 10 client.4556.objecter ms_dispatch 0x7f1978000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:33.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.771+0000 7f196dffb640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:33.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.771+0000 7f196dffb640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:33.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.771+0000 7f196dffb640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-25T19:29:33.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.771+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1968050440 con 0x7f19780a98f0 2024-03-25T19:29:33.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.771+0000 7f196dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:33.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:33.771+0000 7f196dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:34.036 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.036+0000 7f197f3cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:34.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.036+0000 7f197f3cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:34.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.036+0000 7f197f3cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:34.037 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.036+0000 7f197f3cd640 1 -- 172.21.15.123:0/3718300093 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f197813db40 con 0x7f19780a98f0 2024-03-25T19:29:34.070 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.070+0000 7f196dffb640 1 -- 172.21.15.123:0/3718300093 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f1968052370 con 0x7f19780a98f0 2024-03-25T19:29:34.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.071+0000 7f196dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:34.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.071+0000 7f196dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:34.071 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:29:34.071 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:34.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 10 librados: watch_flush enter 2024-03-25T19:29:34.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 10 librados: watch_flush exit 2024-03-25T19:29:34.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-25T19:29:34.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 10 client.4556.objecter successfully canceled tick 2024-03-25T19:29:34.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 1 -- 172.21.15.123:0/3718300093 >> v1:172.21.15.151:6800/727684239 conn(0x7f195c06de00 legacy=0x7f195c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:34.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 10 monclient: shutdown 2024-03-25T19:29:34.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:34.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 1 -- 172.21.15.123:0/3718300093 >> v1:172.21.15.123:6790/0 conn(0x7f19780a98f0 legacy=0x7f1978136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:34.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 1 -- 172.21.15.123:0/3718300093 shutdown_connections 2024-03-25T19:29:34.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 1 -- 172.21.15.123:0/3718300093 >> 172.21.15.123:0/3718300093 conn(0x7f19780b7ea0 msgr2=0x7f1978074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:34.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 1 -- 172.21.15.123:0/3718300093 shutdown_connections 2024-03-25T19:29:34.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 1 -- 172.21.15.123:0/3718300093 wait complete. 2024-03-25T19:29:34.074 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:34.073+0000 7f197f3cd640 1 librados: shutdown 2024-03-25T19:29:34.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:36.021 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:07:25.593 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:36.029 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:07:25.601 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:36.053 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:07:25.625 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:36.079 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:07:25.651 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:36.081 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:07:25.614 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:36.088 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:07:25.622 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:36.091 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:07:25.663 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:36.118 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:07:25.690 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:36.121 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:07:25.655 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:40.893 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:40.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.991+0000 7f8c4f8a1640 10 monclient: get_monmap_and_config 2024-03-25T19:29:40.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.991+0000 7f8c4f8a1640 10 monclient: build_initial_monmap 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.991+0000 7f8c4f8a1640 10 monclient: monmap: 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:29:40.992620+0000 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:29:40.992620+0000 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:29:40.992 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:40.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.992+0000 7f8c4f8a1640 1 Processor -- start 2024-03-25T19:29:40.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.992+0000 7f8c4f8a1640 1 -- start start 2024-03-25T19:29:40.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.992+0000 7f8c4f8a1640 10 monclient: init 2024-03-25T19:29:40.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.993+0000 7f8c4f8a1640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:40.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.993+0000 7f8c4f8a1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:40.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.993+0000 7f8c4f8a1640 10 monclient(hunting): picked mon.noname-a con 0x7f8c480b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:40.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.993+0000 7f8c4f8a1640 10 monclient(hunting): picked mon.noname-c con 0x7f8c480b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:40.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.993+0000 7f8c4f8a1640 10 monclient(hunting): picked mon.noname-b con 0x7f8c4814be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:40.994 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.993+0000 7f8c4f8a1640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c480af940 con 0x7f8c480b4d60 2024-03-25T19:29:40.995 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.993+0000 7f8c4f8a1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c480b0a70 con 0x7f8c480b1730 2024-03-25T19:29:40.995 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.993+0000 7f8c4f8a1640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c480b0d50 con 0x7f8c4814be90 2024-03-25T19:29:40.995 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.993+0000 7f8c4f8a1640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:40.995 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:40.993+0000 7f8c4f8a1640 10 monclient(hunting): authenticate will time out at 6018.353152s 2024-03-25T19:29:41.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.014+0000 7f8c4ce3c640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f8c480b1730 0x7f8c480b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:42376/0 (socket says 172.21.15.123:42376) 2024-03-25T19:29:41.014 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.014+0000 7f8c4ce3c640 1 -- 172.21.15.123:0/2852996048 learned_addr learned my addr 172.21.15.123:0/2852996048 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:41.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.035+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949491169 0 0) 0x7f8c480b0a70 con 0x7f8c480b1730 2024-03-25T19:29:41.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.035+0000 7f8c3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.035+0000 7f8c3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.035+0000 7f8c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.035+0000 7f8c3ffff640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:29:41.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.035+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c34003520 con 0x7f8c480b1730 2024-03-25T19:29:41.051 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.051+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614036444 0 0) 0x7f8c480af940 con 0x7f8c480b4d60 2024-03-25T19:29:41.051 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.051+0000 7f8c3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.051 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.051+0000 7f8c3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.051 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.051+0000 7f8c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.051+0000 7f8c3ffff640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:29:41.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.051+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c480b0a70 con 0x7f8c480b4d60 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4028037815 0 0) 0x7f8c34003520 con 0x7f8c480b1730 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 >> v1:172.21.15.123:6790/0 conn(0x7f8c4814be90 legacy=0x7f8c4814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 >> v1:172.21.15.123:6789/0 conn(0x7f8c480b4d60 legacy=0x7f8c4813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 1 monclient: found mon.noname-c 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c480b1030 con 0x7f8c480b1730 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 10 monclient: _finish_auth 0 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8c44001690 con 0x7f8c480b1730 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:41.057 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c3ffff640 10 monclient: dump: 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c4f8a1640 5 monclient: authenticate success, global_id 4537 2024-03-25T19:29:41.058 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.056+0000 7f8c4f8a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:41.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.089+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c44001870 con 0x7f8c480b1730 2024-03-25T19:29:41.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.089+0000 7f8c3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:41.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.089+0000 7f8c3ffff640 1 -- 172.21.15.123:0/2852996048 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8c44001c80 con 0x7f8c480b1730 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.089+0000 7f8c3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.089+0000 7f8c3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.089+0000 7f8c3ffff640 10 monclient: dump: 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:41.090 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:41.091 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:41.091 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.089+0000 7f8c4f8a1640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:41.091 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.089+0000 7f8c4f8a1640 4 set_mon_vals no callback set 2024-03-25T19:29:41.092 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.090+0000 7f8c4f8a1640 10 monclient: shutdown 2024-03-25T19:29:41.092 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.090+0000 7f8c4f8a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:41.092 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.090+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/2852996048 >> v1:172.21.15.151:6789/0 conn(0x7f8c480b1730 legacy=0x7f8c480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.092 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.090+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/2852996048 shutdown_connections 2024-03-25T19:29:41.092 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.090+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/2852996048 >> 172.21.15.123:0/2852996048 conn(0x7f8c480617a0 msgr2=0x7f8c48061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.091+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/2852996048 shutdown_connections 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.091+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/2852996048 wait complete. 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.091+0000 7f8c4f8a1640 10 monclient: build_initial_monmap 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.091+0000 7f8c4f8a1640 10 monclient: monmap: 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:41.093 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.092+0000 7f8c4f8a1640 1 librados: starting msgr at 2024-03-25T19:29:41.094 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.092+0000 7f8c4f8a1640 1 librados: starting objecter 2024-03-25T19:29:41.094 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.092+0000 7f8c4f8a1640 1 Processor -- start 2024-03-25T19:29:41.094 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.092+0000 7f8c4f8a1640 1 -- start start 2024-03-25T19:29:41.094 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.092+0000 7f8c4f8a1640 1 librados: setting wanted keys 2024-03-25T19:29:41.094 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.092+0000 7f8c4f8a1640 1 librados: calling monclient init 2024-03-25T19:29:41.094 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.092+0000 7f8c4f8a1640 10 monclient: init 2024-03-25T19:29:41.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.093+0000 7f8c4f8a1640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:41.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.093+0000 7f8c4f8a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:41.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.093+0000 7f8c4f8a1640 10 monclient(hunting): picked mon.noname-b con 0x7f8c480b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:41.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.093+0000 7f8c4f8a1640 10 monclient(hunting): picked mon.noname-c con 0x7f8c480b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:41.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.093+0000 7f8c4f8a1640 10 monclient(hunting): picked mon.noname-a con 0x7f8c4814be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:41.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.093+0000 7f8c4f8a1640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c481d7cf0 con 0x7f8c4814be90 2024-03-25T19:29:41.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.093+0000 7f8c4f8a1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c481d8890 con 0x7f8c480b1730 2024-03-25T19:29:41.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.093+0000 7f8c4f8a1640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4813ab70 con 0x7f8c480b4d60 2024-03-25T19:29:41.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.093+0000 7f8c4f8a1640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:41.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.093+0000 7f8c4f8a1640 10 monclient(hunting): authenticate will time out at 6018.453190s 2024-03-25T19:29:41.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.103+0000 7f8c4de3e640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f8c4814be90 0x7f8c481e31c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:48896/0 (socket says 172.21.15.123:48896) 2024-03-25T19:29:41.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.103+0000 7f8c4de3e640 1 -- 172.21.15.123:0/3251597902 learned_addr learned my addr 172.21.15.123:0/3251597902 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:41.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.105+0000 7f8c3dffb640 10 client.?.objecter ms_handle_connect 0x7f8c4814be90 2024-03-25T19:29:41.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.105+0000 7f8c3dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:41.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.111+0000 7f8c3dffb640 10 client.?.objecter ms_handle_connect 0x7f8c480b1730 2024-03-25T19:29:41.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.111+0000 7f8c3dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:41.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.114+0000 7f8c3dffb640 10 client.?.objecter ms_handle_connect 0x7f8c480b4d60 2024-03-25T19:29:41.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.114+0000 7f8c3dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:41.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.130+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471296736 0 0) 0x7f8c481d8890 con 0x7f8c480b1730 2024-03-25T19:29:41.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.130+0000 7f8c3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.130+0000 7f8c3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.130+0000 7f8c3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.130+0000 7f8c3dffb640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:29:41.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.130+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c2c004eb0 con 0x7f8c480b1730 2024-03-25T19:29:41.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.131+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273105171 0 0) 0x7f8c4813ab70 con 0x7f8c480b4d60 2024-03-25T19:29:41.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.131+0000 7f8c3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.131+0000 7f8c3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.131+0000 7f8c3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.131+0000 7f8c3dffb640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:29:41.131 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.131+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c481d8890 con 0x7f8c480b4d60 2024-03-25T19:29:41.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.135+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170109382 0 0) 0x7f8c481d7cf0 con 0x7f8c4814be90 2024-03-25T19:29:41.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.135+0000 7f8c3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.135+0000 7f8c3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.135+0000 7f8c3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.135+0000 7f8c3dffb640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:29:41.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.135+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c4813ab70 con 0x7f8c4814be90 2024-03-25T19:29:41.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.147+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1292286705 0 0) 0x7f8c2c004eb0 con 0x7f8c480b1730 2024-03-25T19:29:41.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.147+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c481d7cf0 con 0x7f8c480b1730 2024-03-25T19:29:41.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.147+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8c38001930 con 0x7f8c480b1730 2024-03-25T19:29:41.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.147+0000 7f8c3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:41.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.147+0000 7f8c3dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.147+0000 7f8c3dffb640 10 monclient(hunting): dump: 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:41.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.147+0000 7f8c3dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:41.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.151+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2561372574 0 0) 0x7f8c481d8890 con 0x7f8c480b4d60 2024-03-25T19:29:41.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.151+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c2c004eb0 con 0x7f8c480b4d60 2024-03-25T19:29:41.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.151+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8c44001b80 con 0x7f8c480b4d60 2024-03-25T19:29:41.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.151+0000 7f8c3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:41.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.151+0000 7f8c3dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:41.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.151+0000 7f8c3dffb640 10 monclient(hunting): dump: 2024-03-25T19:29:41.151 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:41.152 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:41.152 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:41.152 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:41.152 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:41.152 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:41.152 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:41.152 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:41.152 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:41.152 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:41.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 578530439 0 0) 0x7f8c2c004eb0 con 0x7f8c480b4d60 2024-03-25T19:29:41.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 >> v1:172.21.15.151:6789/0 conn(0x7f8c480b1730 legacy=0x7f8c481d8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 >> v1:172.21.15.123:6789/0 conn(0x7f8c4814be90 legacy=0x7f8c481e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:29:41.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 1 monclient: found mon.c 2024-03-25T19:29:41.169 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c481d7980 con 0x7f8c480b4d60 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 10 monclient: _finish_auth 0 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c4f8a1640 5 monclient: authenticate success, global_id 4562 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c4f8a1640 10 monclient: _renew_subs 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c4f8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/3251597902 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c48139980 con 0x7f8c480b4d60 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c4f8a1640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c4f8a1640 10 monclient: _renew_subs 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c4f8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/3251597902 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c48139710 con 0x7f8c480b4d60 2024-03-25T19:29:41.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.169+0000 7f8c4f8a1640 1 librados: init done 2024-03-25T19:29:41.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.170+0000 7f8c4f8a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:41.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.170+0000 7f8c4f8a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:41.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.170+0000 7f8c4f8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:41.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.170+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/3251597902 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c480627b0 con 0x7f8c480b4d60 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.483+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c44001d00 con 0x7f8c480b4d60 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.483+0000 7f8c3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.483+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8c44004d90 con 0x7f8c480b4d60 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.483+0000 7f8c3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.483+0000 7f8c3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.483+0000 7f8c3dffb640 10 monclient: dump: 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:41.485 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:41.485 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:41.485 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:41.485 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:41.485 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:41.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.483+0000 7f8c3f7fe640 4 set_mon_vals no callback set 2024-03-25T19:29:41.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.484+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f8c4401ad60 con 0x7f8c480b4d60 2024-03-25T19:29:41.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.484+0000 7f8c4d63d640 10 monclient: get_auth_request con 0x7f8c2c06dfb0 auth_method 0 2024-03-25T19:29:41.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.485+0000 7f8c3dffb640 10 client.4562.objecter ms_handle_connect 0x7f8c2c06dfb0 2024-03-25T19:29:41.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.489+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f8c440548d0 con 0x7f8c480b4d60 2024-03-25T19:29:41.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.489+0000 7f8c3dffb640 10 client.4562.objecter ms_dispatch 0x7f8c48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:41.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.489+0000 7f8c3dffb640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:41.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.489+0000 7f8c3dffb640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:41.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.489+0000 7f8c3dffb640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-25T19:29:41.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.489+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c440503e0 con 0x7f8c480b4d60 2024-03-25T19:29:41.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.489+0000 7f8c3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:41.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.489+0000 7f8c3dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:41.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.732+0000 7f8c4f8a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:41.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.732+0000 7f8c4f8a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:41.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.732+0000 7f8c4f8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:29:41.733 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.732+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/3251597902 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c48069f40 con 0x7f8c480b4d60 2024-03-25T19:29:41.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.769+0000 7f8c3dffb640 1 -- 172.21.15.123:0/3251597902 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f8c44052310 con 0x7f8c480b4d60 2024-03-25T19:29:41.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.769+0000 7f8c3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:41.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.769+0000 7f8c3dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:41.769 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:29:41.769 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:41.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 10 librados: watch_flush enter 2024-03-25T19:29:41.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 10 librados: watch_flush exit 2024-03-25T19:29:41.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-25T19:29:41.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 10 client.4562.objecter successfully canceled tick 2024-03-25T19:29:41.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/3251597902 >> v1:172.21.15.151:6800/727684239 conn(0x7f8c2c06dfb0 legacy=0x7f8c2c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 10 monclient: shutdown 2024-03-25T19:29:41.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:41.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/3251597902 >> v1:172.21.15.123:6790/0 conn(0x7f8c480b4d60 legacy=0x7f8c4813bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/3251597902 shutdown_connections 2024-03-25T19:29:41.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/3251597902 >> 172.21.15.123:0/3251597902 conn(0x7f8c480617a0 msgr2=0x7f8c480b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:41.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/3251597902 shutdown_connections 2024-03-25T19:29:41.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 1 -- 172.21.15.123:0/3251597902 wait complete. 2024-03-25T19:29:41.772 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:41.771+0000 7f8c4f8a1640 1 librados: shutdown 2024-03-25T19:29:41.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:48.590 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:48.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.708+0000 7f9c6d495640 10 monclient: get_monmap_and_config 2024-03-25T19:29:48.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.708+0000 7f9c6d495640 10 monclient: build_initial_monmap 2024-03-25T19:29:48.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.708+0000 7f9c6d495640 10 monclient: monmap: 2024-03-25T19:29:48.708 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:48.708 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:48.709 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:29:48.709292+0000 2024-03-25T19:29:48.709 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:29:48.709292+0000 2024-03-25T19:29:48.709 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:48.709 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:48.709 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:48.709 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:29:48.709 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:29:48.709 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:48.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.709+0000 7f9c6d495640 1 Processor -- start 2024-03-25T19:29:48.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.709+0000 7f9c6d495640 1 -- start start 2024-03-25T19:29:48.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.709+0000 7f9c6d495640 10 monclient: init 2024-03-25T19:29:48.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.709+0000 7f9c6d495640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:48.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.709+0000 7f9c6d495640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:48.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.709+0000 7f9c6d495640 10 monclient(hunting): picked mon.noname-b con 0x7f9c680a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:48.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.710+0000 7f9c6d495640 10 monclient(hunting): picked mon.noname-c con 0x7f9c6813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:48.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.710+0000 7f9c6d495640 10 monclient(hunting): picked mon.noname-a con 0x7f9c681430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:48.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.710+0000 7f9c6d495640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c68142520 con 0x7f9c681430c0 2024-03-25T19:29:48.711 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.710+0000 7f9c6d495640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c680b28a0 con 0x7f9c6813f710 2024-03-25T19:29:48.711 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.710+0000 7f9c6d495640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c680b2a30 con 0x7f9c680a98f0 2024-03-25T19:29:48.711 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.710+0000 7f9c6d495640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:48.711 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.710+0000 7f9c6d495640 10 monclient(hunting): authenticate will time out at 6026.070006s 2024-03-25T19:29:48.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.713+0000 7f9c66ffd640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f9c680a98f0 0x7f9c680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:34326/0 (socket says 172.21.15.123:34326) 2024-03-25T19:29:48.713 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.713+0000 7f9c66ffd640 1 -- 172.21.15.123:0/3357823649 learned_addr learned my addr 172.21.15.123:0/3357823649 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:48.720 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.720+0000 7f9c667fc640 1 -- 172.21.15.123:0/3357823649 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184369187 0 0) 0x7f9c680b2a30 con 0x7f9c680a98f0 2024-03-25T19:29:48.721 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.721+0000 7f9c667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:48.721 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.721+0000 7f9c667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:48.721 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.721+0000 7f9c667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:48.721 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.721+0000 7f9c667fc640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:29:48.721 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.721+0000 7f9c667fc640 1 -- 172.21.15.123:0/3357823649 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c54003520 con 0x7f9c680a98f0 2024-03-25T19:29:48.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 1 -- 172.21.15.123:0/3357823649 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1250386215 0 0) 0x7f9c54003520 con 0x7f9c680a98f0 2024-03-25T19:29:48.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 1 -- 172.21.15.123:0/3357823649 >> v1:172.21.15.151:6789/0 conn(0x7f9c6813f710 legacy=0x7f9c681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 1 -- 172.21.15.123:0/3357823649 >> v1:172.21.15.123:6789/0 conn(0x7f9c681430c0 legacy=0x7f9c6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 10 monclient: _finish_hunting 0 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 1 monclient: found mon.noname-b 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 1 -- 172.21.15.123:0/3357823649 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c680b44c0 con 0x7f9c680a98f0 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 10 monclient: _finish_auth 0 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 10 monclient: _check_auth_tickets 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 1 -- 172.21.15.123:0/3357823649 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9c58001690 con 0x7f9c680a98f0 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c667fc640 10 monclient: dump: 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:48.731 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:48.732 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:48.732 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:48.732 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:48.732 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:48.732 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:48.732 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:48.732 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:48.732 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:48.732 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c6d495640 5 monclient: authenticate success, global_id 4565 2024-03-25T19:29:48.732 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.730+0000 7f9c6d495640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:48.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.741+0000 7f9c667fc640 1 -- 172.21.15.123:0/3357823649 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c580017f0 con 0x7f9c680a98f0 2024-03-25T19:29:48.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.741+0000 7f9c667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:48.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.741+0000 7f9c667fc640 1 -- 172.21.15.123:0/3357823649 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9c58001c00 con 0x7f9c680a98f0 2024-03-25T19:29:48.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.741+0000 7f9c667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:48.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.741+0000 7f9c667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:48.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.741+0000 7f9c667fc640 10 monclient: dump: 2024-03-25T19:29:48.741 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:48.741 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:48.741 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:48.741 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:48.742 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:48.742 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:48.742 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:48.742 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:48.742 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:48.742 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:48.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.741+0000 7f9c6d495640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:48.742 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.741+0000 7f9c6d495640 4 set_mon_vals no callback set 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.742+0000 7f9c6d495640 10 monclient: shutdown 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.742+0000 7f9c6d495640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.742+0000 7f9c6d495640 1 -- 172.21.15.123:0/3357823649 >> v1:172.21.15.123:6790/0 conn(0x7f9c680a98f0 legacy=0x7f9c680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.742+0000 7f9c6d495640 1 -- 172.21.15.123:0/3357823649 shutdown_connections 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.742+0000 7f9c6d495640 1 -- 172.21.15.123:0/3357823649 >> 172.21.15.123:0/3357823649 conn(0x7f9c680b7ea0 msgr2=0x7f9c680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.742+0000 7f9c6d495640 1 -- 172.21.15.123:0/3357823649 shutdown_connections 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.742+0000 7f9c6d495640 1 -- 172.21.15.123:0/3357823649 wait complete. 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.743+0000 7f9c6d495640 10 monclient: build_initial_monmap 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.743+0000 7f9c6d495640 10 monclient: monmap: 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:29:48.745 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.743+0000 7f9c6d495640 1 librados: starting msgr at 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.743+0000 7f9c6d495640 1 librados: starting objecter 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.743+0000 7f9c6d495640 1 Processor -- start 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 1 -- start start 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 1 librados: setting wanted keys 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 1 librados: calling monclient init 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 10 monclient: init 2024-03-25T19:29:48.746 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:48.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:48.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 10 monclient(hunting): picked mon.noname-a con 0x7f9c680a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:48.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 10 monclient(hunting): picked mon.noname-c con 0x7f9c6813f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:48.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 10 monclient(hunting): picked mon.noname-b con 0x7f9c681430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:48.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c68135730 con 0x7f9c680a98f0 2024-03-25T19:29:48.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c681358c0 con 0x7f9c681430c0 2024-03-25T19:29:48.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c68135a50 con 0x7f9c6813f710 2024-03-25T19:29:48.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:48.747 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.744+0000 7f9c6d495640 10 monclient(hunting): authenticate will time out at 6026.104860s 2024-03-25T19:29:48.748 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.749+0000 7f9c66ffd640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f9c680a98f0 0x7f9c681dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:33620/0 (socket says 172.21.15.123:33620) 2024-03-25T19:29:48.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.749+0000 7f9c66ffd640 1 -- 172.21.15.123:0/283262945 learned_addr learned my addr 172.21.15.123:0/283262945 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:48.753 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.753+0000 7f9c5ffff640 10 client.?.objecter ms_handle_connect 0x7f9c681430c0 2024-03-25T19:29:48.753 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.753+0000 7f9c5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:48.753 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.753+0000 7f9c5ffff640 10 client.?.objecter ms_handle_connect 0x7f9c680a98f0 2024-03-25T19:29:48.753 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.753+0000 7f9c5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:48.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.758+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466779742 0 0) 0x7f9c68135730 con 0x7f9c680a98f0 2024-03-25T19:29:48.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.758+0000 7f9c5ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:48.759 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.758+0000 7f9c5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:48.759 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.758+0000 7f9c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:48.759 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.758+0000 7f9c5ffff640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:29:48.759 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.758+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c48004eb0 con 0x7f9c680a98f0 2024-03-25T19:29:48.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.770+0000 7f9c5ffff640 10 client.?.objecter ms_handle_connect 0x7f9c6813f710 2024-03-25T19:29:48.770 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.770+0000 7f9c5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:48.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.771+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733657388 0 0) 0x7f9c681358c0 con 0x7f9c681430c0 2024-03-25T19:29:48.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.771+0000 7f9c5ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:48.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.771+0000 7f9c5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:48.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.771+0000 7f9c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:48.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.771+0000 7f9c5ffff640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:29:48.771 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.771+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c68135730 con 0x7f9c681430c0 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.780+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2830785312 0 0) 0x7f9c68135730 con 0x7f9c681430c0 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.780+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c681358c0 con 0x7f9c681430c0 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.780+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9c60001b00 con 0x7f9c681430c0 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.780+0000 7f9c5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.780+0000 7f9c5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.780+0000 7f9c5ffff640 10 monclient(hunting): dump: 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:48.780 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:48.781 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:48.781 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:48.781 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:48.781 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:48.781 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.780+0000 7f9c5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:48.783 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.783+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205573033 0 0) 0x7f9c68135a50 con 0x7f9c6813f710 2024-03-25T19:29:48.783 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.783+0000 7f9c5ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:48.783 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.783+0000 7f9c5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:48.783 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.783+0000 7f9c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:48.783 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.783+0000 7f9c5ffff640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:29:48.783 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.783+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c68135730 con 0x7f9c6813f710 2024-03-25T19:29:48.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3976542600 0 0) 0x7f9c681358c0 con 0x7f9c681430c0 2024-03-25T19:29:48.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 >> v1:172.21.15.123:6790/0 conn(0x7f9c6813f710 legacy=0x7f9c681d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 >> v1:172.21.15.123:6789/0 conn(0x7f9c680a98f0 legacy=0x7f9c681dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 1 monclient: found mon.b 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c681d5000 con 0x7f9c681430c0 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 10 monclient: _finish_auth 0 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c6d495640 5 monclient: authenticate success, global_id 4546 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c6d495640 10 monclient: _renew_subs 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c6d495640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c6d495640 1 -- 172.21.15.123:0/283262945 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c681d52b0 con 0x7f9c681430c0 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c6d495640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:48.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c6d495640 10 monclient: _renew_subs 2024-03-25T19:29:48.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c6d495640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:29:48.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c6d495640 1 -- 172.21.15.123:0/283262945 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6813c900 con 0x7f9c681430c0 2024-03-25T19:29:48.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.791+0000 7f9c6d495640 1 librados: init done 2024-03-25T19:29:48.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.792+0000 7f9c6d495640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:48.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.793+0000 7f9c6d495640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:48.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.793+0000 7f9c6d495640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:29:48.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:48.793+0000 7f9c6d495640 1 -- 172.21.15.123:0/283262945 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6813cbb0 con 0x7f9c681430c0 2024-03-25T19:29:49.097 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.097+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c60001fb0 con 0x7f9c681430c0 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.097+0000 7f9c5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.097+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9c60003970 con 0x7f9c681430c0 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.097+0000 7f9c5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.097+0000 7f9c5ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.097+0000 7f9c5ffff640 10 monclient: dump: 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:49.098 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.097+0000 7f9c65ffb640 4 set_mon_vals no callback set 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.098+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f9c60003b90 con 0x7f9c681430c0 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.099+0000 7f9c66ffd640 10 monclient: get_auth_request con 0x7f9c4806de10 auth_method 0 2024-03-25T19:29:49.100 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.099+0000 7f9c5ffff640 10 client.4546.objecter ms_handle_connect 0x7f9c4806de10 2024-03-25T19:29:49.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.104+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f9c60053560 con 0x7f9c681430c0 2024-03-25T19:29:49.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.104+0000 7f9c5ffff640 10 client.4546.objecter ms_dispatch 0x7f9c68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:49.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.104+0000 7f9c5ffff640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:49.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.104+0000 7f9c5ffff640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:49.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.104+0000 7f9c5ffff640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T19:29:49.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.104+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c6004f070 con 0x7f9c681430c0 2024-03-25T19:29:49.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.104+0000 7f9c5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:49.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.104+0000 7f9c5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:49.322 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.322+0000 7f9c6d495640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:49.323 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.322+0000 7f9c6d495640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:49.323 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.322+0000 7f9c6d495640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:29:49.323 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.322+0000 7f9c6d495640 1 -- 172.21.15.123:0/283262945 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c681371c0 con 0x7f9c681430c0 2024-03-25T19:29:49.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.414+0000 7f9c5ffff640 1 -- 172.21.15.123:0/283262945 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f9c60050fa0 con 0x7f9c681430c0 2024-03-25T19:29:49.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.414+0000 7f9c5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:49.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.414+0000 7f9c5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:49.414 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:29:49.415 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:49.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 10 librados: watch_flush enter 2024-03-25T19:29:49.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 10 librados: watch_flush exit 2024-03-25T19:29:49.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T19:29:49.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 10 client.4546.objecter successfully canceled tick 2024-03-25T19:29:49.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 1 -- 172.21.15.123:0/283262945 >> v1:172.21.15.151:6800/727684239 conn(0x7f9c4806de10 legacy=0x7f9c480702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 10 monclient: shutdown 2024-03-25T19:29:49.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:49.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 1 -- 172.21.15.123:0/283262945 >> v1:172.21.15.151:6789/0 conn(0x7f9c681430c0 legacy=0x7f9c681d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 1 -- 172.21.15.123:0/283262945 shutdown_connections 2024-03-25T19:29:49.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 1 -- 172.21.15.123:0/283262945 >> 172.21.15.123:0/283262945 conn(0x7f9c680b7ea0 msgr2=0x7f9c6814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:49.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 1 -- 172.21.15.123:0/283262945 shutdown_connections 2024-03-25T19:29:49.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.416+0000 7f9c6d495640 1 -- 172.21.15.123:0/283262945 wait complete. 2024-03-25T19:29:49.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:49.417+0000 7f9c6d495640 1 librados: shutdown 2024-03-25T19:29:49.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:56.115 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:07:45.687 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:56.121 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:07:45.693 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:56.157 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:07:45.729 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:56.175 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:07:45.747 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:56.179 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:07:45.713 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:56.183 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:07:45.716 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:56.186 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:07:45.758 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:56.212 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:07:45.745 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:56.218 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:07:45.790 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:56.234 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:56.342 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.342+0000 7f0e8fd3b640 10 monclient: get_monmap_and_config 2024-03-25T19:29:56.342 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.342+0000 7f0e8fd3b640 10 monclient: build_initial_monmap 2024-03-25T19:29:56.342 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.342+0000 7f0e8fd3b640 10 monclient: monmap: 2024-03-25T19:29:56.343 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:56.343 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:56.343 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:29:56.343200+0000 2024-03-25T19:29:56.343 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:29:56.343200+0000 2024-03-25T19:29:56.343 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:56.343 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:56.343 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:56.343 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:29:56.344 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:29:56.344 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:56.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.342+0000 7f0e8fd3b640 1 Processor -- start 2024-03-25T19:29:56.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 1 -- start start 2024-03-25T19:29:56.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 10 monclient: init 2024-03-25T19:29:56.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:56.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:56.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 10 monclient(hunting): picked mon.noname-a con 0x7f0e880a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:56.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 10 monclient(hunting): picked mon.noname-c con 0x7f0e8813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:56.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 10 monclient(hunting): picked mon.noname-b con 0x7f0e881430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:56.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e88142520 con 0x7f0e880a98f0 2024-03-25T19:29:56.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e880b28a0 con 0x7f0e8813f710 2024-03-25T19:29:56.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e880b2a30 con 0x7f0e881430c0 2024-03-25T19:29:56.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:56.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.343+0000 7f0e8fd3b640 10 monclient(hunting): authenticate will time out at 6033.703712s 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.369+0000 7f0e8dad7640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f0e880a98f0 0x7f0e880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46660/0 (socket says 172.21.15.123:46660) 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.369+0000 7f0e8dad7640 1 -- 172.21.15.123:0/3654686943 learned_addr learned my addr 172.21.15.123:0/3654686943 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:56.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.395+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850271436 0 0) 0x7f0e88142520 con 0x7f0e880a98f0 2024-03-25T19:29:56.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.395+0000 7f0e8cad5640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.395+0000 7f0e8cad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.395+0000 7f0e8cad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.395+0000 7f0e8cad5640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:29:56.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.395+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e74003520 con 0x7f0e880a98f0 2024-03-25T19:29:56.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.407+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907332091 0 0) 0x7f0e880b2a30 con 0x7f0e881430c0 2024-03-25T19:29:56.406 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.407+0000 7f0e8cad5640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.407 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.407+0000 7f0e8cad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.407 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.407+0000 7f0e8cad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.407 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.407+0000 7f0e8cad5640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:29:56.407 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.407+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e88142520 con 0x7f0e881430c0 2024-03-25T19:29:56.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.414+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516771174 0 0) 0x7f0e880b28a0 con 0x7f0e8813f710 2024-03-25T19:29:56.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.414+0000 7f0e8cad5640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.414+0000 7f0e8cad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.414+0000 7f0e8cad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.414+0000 7f0e8cad5640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:29:56.415 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.414+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e880b2a30 con 0x7f0e8813f710 2024-03-25T19:29:56.425 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 975182162 0 0) 0x7f0e74003520 con 0x7f0e880a98f0 2024-03-25T19:29:56.425 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 >> v1:172.21.15.123:6790/0 conn(0x7f0e881430c0 legacy=0x7f0e8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.426 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 >> v1:172.21.15.151:6789/0 conn(0x7f0e8813f710 legacy=0x7f0e881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.426 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 10 monclient: _finish_hunting 0 2024-03-25T19:29:56.426 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 1 monclient: found mon.noname-a 2024-03-25T19:29:56.426 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:56.426 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:29:56.426 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e880b44c0 con 0x7f0e880a98f0 2024-03-25T19:29:56.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 10 monclient: _finish_auth 0 2024-03-25T19:29:56.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 10 monclient: _check_auth_tickets 2024-03-25T19:29:56.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:56.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0e84001690 con 0x7f0e880a98f0 2024-03-25T19:29:56.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8fd3b640 5 monclient: authenticate success, global_id 4539 2024-03-25T19:29:56.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8fd3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:56.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:56.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8cad5640 10 monclient: dump: 2024-03-25T19:29:56.428 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:56.428 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:56.428 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:56.429 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:56.429 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.429 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:56.429 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:56.429 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:56.429 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:56.429 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:56.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.425+0000 7f0e8fd3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:56.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e840017f0 con 0x7f0e880a98f0 2024-03-25T19:29:56.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8cad5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:56.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8cad5640 1 -- 172.21.15.123:0/3654686943 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0e84001c00 con 0x7f0e880a98f0 2024-03-25T19:29:56.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8fd3b640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:56.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8fd3b640 4 set_mon_vals no callback set 2024-03-25T19:29:56.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8cad5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8cad5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:56.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8cad5640 10 monclient: dump: 2024-03-25T19:29:56.441 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:56.442 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:56.442 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:56.442 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:56.442 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.442 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:56.442 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:56.442 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:56.442 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:56.442 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:56.444 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8fd3b640 10 monclient: shutdown 2024-03-25T19:29:56.444 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8fd3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:56.444 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/3654686943 >> v1:172.21.15.123:6789/0 conn(0x7f0e880a98f0 legacy=0x7f0e880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.444 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/3654686943 shutdown_connections 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.441+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/3654686943 >> 172.21.15.123:0/3654686943 conn(0x7f0e880b7ea0 msgr2=0x7f0e880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.442+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/3654686943 shutdown_connections 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.442+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/3654686943 wait complete. 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.442+0000 7f0e8fd3b640 10 monclient: build_initial_monmap 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.442+0000 7f0e8fd3b640 10 monclient: monmap: 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:29:56.445 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 1 librados: starting msgr at 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 1 librados: starting objecter 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 1 Processor -- start 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 1 -- start start 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 1 librados: setting wanted keys 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 1 librados: calling monclient init 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 10 monclient: init 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 10 monclient(hunting): picked mon.noname-b con 0x7f0e880a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 10 monclient(hunting): picked mon.noname-c con 0x7f0e8813f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 10 monclient(hunting): picked mon.noname-a con 0x7f0e881430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:29:56.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e880b7ce0 con 0x7f0e881430c0 2024-03-25T19:29:56.447 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e881319b0 con 0x7f0e880a98f0 2024-03-25T19:29:56.447 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e880b7800 con 0x7f0e8813f710 2024-03-25T19:29:56.447 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:56.447 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.443+0000 7f0e8fd3b640 10 monclient(hunting): authenticate will time out at 6033.803775s 2024-03-25T19:29:56.452 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.452+0000 7f0e8e2d8640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f0e881430c0 0x7f0e881312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46708/0 (socket says 172.21.15.123:46708) 2024-03-25T19:29:56.452 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.452+0000 7f0e8e2d8640 1 -- 172.21.15.123:0/1617948984 learned_addr learned my addr 172.21.15.123:0/1617948984 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:29:56.453 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.454+0000 7f0e7e7fc640 10 client.?.objecter ms_handle_connect 0x7f0e881430c0 2024-03-25T19:29:56.453 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.454+0000 7f0e7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.463+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793638414 0 0) 0x7f0e880b7ce0 con 0x7f0e881430c0 2024-03-25T19:29:56.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.463+0000 7f0e7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.463+0000 7f0e7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.463+0000 7f0e7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.463+0000 7f0e7e7fc640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:29:56.464 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.463+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e6c004eb0 con 0x7f0e881430c0 2024-03-25T19:29:56.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.467+0000 7f0e7e7fc640 10 client.?.objecter ms_handle_connect 0x7f0e880a98f0 2024-03-25T19:29:56.467 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.467+0000 7f0e7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.476+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1848742438 0 0) 0x7f0e6c004eb0 con 0x7f0e881430c0 2024-03-25T19:29:56.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.476+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e880b7ce0 con 0x7f0e881430c0 2024-03-25T19:29:56.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.476+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0e80001c90 con 0x7f0e881430c0 2024-03-25T19:29:56.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.476+0000 7f0e7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.476+0000 7f0e7e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:56.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.476+0000 7f0e7e7fc640 10 monclient(hunting): dump: 2024-03-25T19:29:56.476 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:56.476 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:56.476 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:56.476 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:56.477 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.477 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:56.477 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:56.477 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:56.477 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:56.477 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:56.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.476+0000 7f0e7e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:56.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.480+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194675524 0 0) 0x7f0e881319b0 con 0x7f0e880a98f0 2024-03-25T19:29:56.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.480+0000 7f0e7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.480+0000 7f0e7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.480+0000 7f0e7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.480+0000 7f0e7e7fc640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:29:56.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.480+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e6c004eb0 con 0x7f0e880a98f0 2024-03-25T19:29:56.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.486+0000 7f0e7e7fc640 10 client.?.objecter ms_handle_connect 0x7f0e8813f710 2024-03-25T19:29:56.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.486+0000 7f0e7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3754366279 0 0) 0x7f0e880b7ce0 con 0x7f0e881430c0 2024-03-25T19:29:56.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 >> v1:172.21.15.123:6790/0 conn(0x7f0e8813f710 legacy=0x7f0e88130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 >> v1:172.21.15.151:6789/0 conn(0x7f0e880a98f0 legacy=0x7f0e881dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:29:56.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 1 monclient: found mon.a 2024-03-25T19:29:56.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:56.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:29:56.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e88136f50 con 0x7f0e881430c0 2024-03-25T19:29:56.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:29:56.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:29:56.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:56.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e8fd3b640 5 monclient: authenticate success, global_id 4542 2024-03-25T19:29:56.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e8fd3b640 10 monclient: _renew_subs 2024-03-25T19:29:56.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e8fd3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:29:56.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.488+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/1617948984 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e881350f0 con 0x7f0e881430c0 2024-03-25T19:29:56.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.489+0000 7f0e8fd3b640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:56.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.489+0000 7f0e8fd3b640 10 monclient: _renew_subs 2024-03-25T19:29:56.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.489+0000 7f0e8fd3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:29:56.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.489+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/1617948984 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e88135c60 con 0x7f0e881430c0 2024-03-25T19:29:56.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.489+0000 7f0e8fd3b640 1 librados: init done 2024-03-25T19:29:56.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.490+0000 7f0e8fd3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:56.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.490+0000 7f0e8fd3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:56.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.490+0000 7f0e8fd3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:29:56.491 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.490+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/1617948984 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e880627b0 con 0x7f0e881430c0 2024-03-25T19:29:56.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.794+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e80003970 con 0x7f0e881430c0 2024-03-25T19:29:56.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.794+0000 7f0e7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:56.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.794+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f0e80001df0 con 0x7f0e881430c0 2024-03-25T19:29:56.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.794+0000 7f0e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.794+0000 7f0e7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.794+0000 7f0e7e7fc640 10 monclient: dump: 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:29:56.795 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.794+0000 7f0e7ffff640 4 set_mon_vals no callback set 2024-03-25T19:29:56.796 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.795+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f0e80003e00 con 0x7f0e881430c0 2024-03-25T19:29:56.796 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.796+0000 7f0e8dad7640 10 monclient: get_auth_request con 0x7f0e6c06dd90 auth_method 0 2024-03-25T19:29:56.796 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.796+0000 7f0e7e7fc640 10 client.4542.objecter ms_handle_connect 0x7f0e6c06dd90 2024-03-25T19:29:56.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.801+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f0e80054ac0 con 0x7f0e881430c0 2024-03-25T19:29:56.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.801+0000 7f0e7e7fc640 10 client.4542.objecter ms_dispatch 0x7f0e88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:56.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.801+0000 7f0e7e7fc640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:56.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.801+0000 7f0e7e7fc640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:56.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.801+0000 7f0e7e7fc640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-25T19:29:56.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.801+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e800505d0 con 0x7f0e881430c0 2024-03-25T19:29:56.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.801+0000 7f0e7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:56.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:56.801+0000 7f0e7e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:57.060 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.060+0000 7f0e8fd3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:57.060 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.060+0000 7f0e8fd3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:57.060 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.060+0000 7f0e8fd3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:29:57.060 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.060+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/1617948984 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e880b7a00 con 0x7f0e881430c0 2024-03-25T19:29:57.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.083+0000 7f0e7e7fc640 1 -- 172.21.15.123:0/1617948984 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f0e880b7a00 con 0x7f0e881430c0 2024-03-25T19:29:57.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.083+0000 7f0e7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:57.083 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.083+0000 7f0e7e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:57.083 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:29:57.084 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:57.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 10 librados: watch_flush enter 2024-03-25T19:29:57.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 10 librados: watch_flush exit 2024-03-25T19:29:57.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-25T19:29:57.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 10 client.4542.objecter successfully canceled tick 2024-03-25T19:29:57.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/1617948984 >> v1:172.21.15.151:6800/727684239 conn(0x7f0e6c06dd90 legacy=0x7f0e6c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:57.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 10 monclient: shutdown 2024-03-25T19:29:57.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:57.087 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/1617948984 >> v1:172.21.15.123:6789/0 conn(0x7f0e881430c0 legacy=0x7f0e881312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:57.087 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/1617948984 shutdown_connections 2024-03-25T19:29:57.087 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/1617948984 >> 172.21.15.123:0/1617948984 conn(0x7f0e880b7ea0 msgr2=0x7f0e88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:57.087 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/1617948984 shutdown_connections 2024-03-25T19:29:57.087 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 1 -- 172.21.15.123:0/1617948984 wait complete. 2024-03-25T19:29:57.087 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:29:57.086+0000 7f0e8fd3b640 1 librados: shutdown 2024-03-25T19:29:57.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:03.904 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:04.022 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.019+0000 7fa5c2f82640 10 monclient: get_monmap_and_config 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.019+0000 7fa5c2f82640 10 monclient: build_initial_monmap 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.019+0000 7fa5c2f82640 10 monclient: monmap: 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:30:04.021121+0000 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:30:04.021121+0000 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:04.023 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.020+0000 7fa5c2f82640 1 Processor -- start 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.020+0000 7fa5c2f82640 1 -- start start 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.020+0000 7fa5c2f82640 10 monclient: init 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.021+0000 7fa5c2f82640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.021+0000 7fa5c2f82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.021+0000 7fa5c2f82640 10 monclient(hunting): picked mon.noname-b con 0x7fa5bc0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.021+0000 7fa5c2f82640 10 monclient(hunting): picked mon.noname-c con 0x7fa5bc0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.021+0000 7fa5c2f82640 10 monclient(hunting): picked mon.noname-a con 0x7fa5bc14be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.021+0000 7fa5c2f82640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0af940 con 0x7fa5bc14be90 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.021+0000 7fa5c2f82640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0b0a70 con 0x7fa5bc0b1730 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.021+0000 7fa5c2f82640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0b0d50 con 0x7fa5bc0b4d60 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.021+0000 7fa5c2f82640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:04.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.021+0000 7fa5c2f82640 10 monclient(hunting): authenticate will time out at 6041.382034s 2024-03-25T19:30:04.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.026+0000 7fa5b3fff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fa5bc0b1730 0x7fa5bc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:59968/0 (socket says 172.21.15.123:59968) 2024-03-25T19:30:04.027 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.026+0000 7fa5b3fff640 1 -- 172.21.15.123:0/181001245 learned_addr learned my addr 172.21.15.123:0/181001245 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.034+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358567137 0 0) 0x7fa5bc0b0a70 con 0x7fa5bc0b1730 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.034+0000 7fa5b37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.034+0000 7fa5b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.034+0000 7fa5b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.034+0000 7fa5b37fe640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.034+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a4003520 con 0x7fa5bc0b1730 2024-03-25T19:30:04.047 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.046+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008236486 0 0) 0x7fa5bc0b0d50 con 0x7fa5bc0b4d60 2024-03-25T19:30:04.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.047+0000 7fa5b37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.047+0000 7fa5b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.047+0000 7fa5b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.047+0000 7fa5b37fe640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:30:04.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.047+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5bc0b0a70 con 0x7fa5bc0b4d60 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4201560492 0 0) 0x7fa5a4003520 con 0x7fa5bc0b1730 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 >> v1:172.21.15.123:6790/0 conn(0x7fa5bc0b4d60 legacy=0x7fa5bc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 >> v1:172.21.15.123:6789/0 conn(0x7fa5bc14be90 legacy=0x7fa5bc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 1 monclient: found mon.noname-c 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc0b1030 con 0x7fa5bc0b1730 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa5ac001690 con 0x7fa5bc0b1730 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5c2f82640 5 monclient: authenticate success, global_id 4555 2024-03-25T19:30:04.063 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5c2f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5b37fe640 10 monclient: dump: 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.061+0000 7fa5c2f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:04.068 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.067+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5ac0017f0 con 0x7fa5bc0b1730 2024-03-25T19:30:04.068 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.067+0000 7fa5b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5c2f82640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5b37fe640 1 -- 172.21.15.123:0/181001245 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa5ac001c40 con 0x7fa5bc0b1730 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5c2f82640 4 set_mon_vals no callback set 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5b37fe640 10 monclient: dump: 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:04.069 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5c2f82640 10 monclient: shutdown 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5c2f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5c2f82640 1 -- 172.21.15.123:0/181001245 >> v1:172.21.15.151:6789/0 conn(0x7fa5bc0b1730 legacy=0x7fa5bc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5c2f82640 1 -- 172.21.15.123:0/181001245 shutdown_connections 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.068+0000 7fa5c2f82640 1 -- 172.21.15.123:0/181001245 >> 172.21.15.123:0/181001245 conn(0x7fa5bc0617a0 msgr2=0x7fa5bc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.069+0000 7fa5c2f82640 1 -- 172.21.15.123:0/181001245 shutdown_connections 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.069+0000 7fa5c2f82640 1 -- 172.21.15.123:0/181001245 wait complete. 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.069+0000 7fa5c2f82640 10 monclient: build_initial_monmap 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.069+0000 7fa5c2f82640 10 monclient: monmap: 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.070+0000 7fa5c2f82640 1 librados: starting msgr at 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.070+0000 7fa5c2f82640 1 librados: starting objecter 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.070+0000 7fa5c2f82640 1 Processor -- start 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.070+0000 7fa5c2f82640 1 -- start start 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.070+0000 7fa5c2f82640 1 librados: setting wanted keys 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.070+0000 7fa5c2f82640 1 librados: calling monclient init 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.070+0000 7fa5c2f82640 10 monclient: init 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.071+0000 7fa5c2f82640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.071+0000 7fa5c2f82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.071+0000 7fa5c2f82640 10 monclient(hunting): picked mon.noname-b con 0x7fa5bc0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.071+0000 7fa5c2f82640 10 monclient(hunting): picked mon.noname-c con 0x7fa5bc0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.071+0000 7fa5c2f82640 10 monclient(hunting): picked mon.noname-a con 0x7fa5bc14be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.071+0000 7fa5c2f82640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0aa790 con 0x7fa5bc14be90 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.071+0000 7fa5c2f82640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc142500 con 0x7fa5bc0b1730 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.071+0000 7fa5c2f82640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc142690 con 0x7fa5bc0b4d60 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.071+0000 7fa5c2f82640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.071+0000 7fa5c2f82640 10 monclient(hunting): authenticate will time out at 6041.431978s 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.079+0000 7fa5c0d1e640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fa5bc0b1730 0x7fa5bc1d7690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:60012/0 (socket says 172.21.15.123:60012) 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.079+0000 7fa5c0d1e640 1 -- 172.21.15.123:0/1051996244 learned_addr learned my addr 172.21.15.123:0/1051996244 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:04.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.085+0000 7fa5b17fa640 10 client.?.objecter ms_handle_connect 0x7fa5bc0b1730 2024-03-25T19:30:04.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.085+0000 7fa5b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:04.094 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.094+0000 7fa5b17fa640 10 client.?.objecter ms_handle_connect 0x7fa5bc14be90 2024-03-25T19:30:04.095 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.094+0000 7fa5b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:04.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.097+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246184426 0 0) 0x7fa5bc142500 con 0x7fa5bc0b1730 2024-03-25T19:30:04.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.097+0000 7fa5b17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.097+0000 7fa5b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.097+0000 7fa5b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.097+0000 7fa5b17fa640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:30:04.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.097+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a8004eb0 con 0x7fa5bc0b1730 2024-03-25T19:30:04.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.097+0000 7fa5b17fa640 10 client.?.objecter ms_handle_connect 0x7fa5bc0b4d60 2024-03-25T19:30:04.099 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.097+0000 7fa5b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:04.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.102+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143847746 0 0) 0x7fa5bc0aa790 con 0x7fa5bc14be90 2024-03-25T19:30:04.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.102+0000 7fa5b17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.102+0000 7fa5b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.102+0000 7fa5b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.102+0000 7fa5b17fa640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:30:04.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.102+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5bc142500 con 0x7fa5bc14be90 2024-03-25T19:30:04.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.108+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602912256 0 0) 0x7fa5bc142690 con 0x7fa5bc0b4d60 2024-03-25T19:30:04.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.108+0000 7fa5b17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.110 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.108+0000 7fa5b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.110 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.108+0000 7fa5b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.110 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.108+0000 7fa5b17fa640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:30:04.110 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.109+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5bc0aa790 con 0x7fa5bc0b4d60 2024-03-25T19:30:04.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.112+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3629900507 0 0) 0x7fa5a8004eb0 con 0x7fa5bc0b1730 2024-03-25T19:30:04.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.112+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5bc142690 con 0x7fa5bc0b1730 2024-03-25T19:30:04.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.112+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa5b8001a10 con 0x7fa5bc0b1730 2024-03-25T19:30:04.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.112+0000 7fa5b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.112+0000 7fa5b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:04.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.112+0000 7fa5b17fa640 10 monclient(hunting): dump: 2024-03-25T19:30:04.114 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:04.114 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:04.114 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:04.114 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.112+0000 7fa5b17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.113+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4043773054 0 0) 0x7fa5bc142500 con 0x7fa5bc14be90 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.113+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5a8004eb0 con 0x7fa5bc14be90 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.113+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa5b4001b90 con 0x7fa5bc14be90 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.113+0000 7fa5b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.113+0000 7fa5b17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.113+0000 7fa5b17fa640 10 monclient(hunting): dump: 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:04.115 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:04.116 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:04.116 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:04.116 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.116 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:04.116 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:04.116 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:04.116 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:04.116 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:04.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 227269303 0 0) 0x7fa5bc142690 con 0x7fa5bc0b1730 2024-03-25T19:30:04.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 >> v1:172.21.15.123:6790/0 conn(0x7fa5bc0b4d60 legacy=0x7fa5bc1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 >> v1:172.21.15.123:6789/0 conn(0x7fa5bc14be90 legacy=0x7fa5bc1e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 1 monclient: found mon.b 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc13b670 con 0x7fa5bc0b1730 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 10 monclient: _finish_auth 0 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5c2f82640 5 monclient: authenticate success, global_id 4558 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5c2f82640 10 monclient: _renew_subs 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5c2f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5c2f82640 1 -- 172.21.15.123:0/1051996244 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc13b920 con 0x7fa5bc0b1730 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5c2f82640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5c2f82640 10 monclient: _renew_subs 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5c2f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:04.120 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5c2f82640 1 -- 172.21.15.123:0/1051996244 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc143220 con 0x7fa5bc0b1730 2024-03-25T19:30:04.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.118+0000 7fa5c2f82640 1 librados: init done 2024-03-25T19:30:04.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.119+0000 7fa5c2f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:04.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.119+0000 7fa5c2f82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:04.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.119+0000 7fa5c2f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:04.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.119+0000 7fa5c2f82640 1 -- 172.21.15.123:0/1051996244 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc0a9a10 con 0x7fa5bc0b1730 2024-03-25T19:30:04.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.395+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5b8001c50 con 0x7fa5bc0b1730 2024-03-25T19:30:04.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.395+0000 7fa5b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:04.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.395+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fa5b8003a60 con 0x7fa5bc0b1730 2024-03-25T19:30:04.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.395+0000 7fa5b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.395+0000 7fa5b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:04.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.395+0000 7fa5b17fa640 10 monclient: dump: 2024-03-25T19:30:04.396 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:04.396 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:04.397 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:04.397 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:04.397 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.397 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:04.397 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:04.397 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:04.397 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:04.397 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:04.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.395+0000 7fa5b2ffd640 4 set_mon_vals no callback set 2024-03-25T19:30:04.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.396+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fa5b8003c80 con 0x7fa5bc0b1730 2024-03-25T19:30:04.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.396+0000 7fa5b3fff640 10 monclient: get_auth_request con 0x7fa5a806df70 auth_method 0 2024-03-25T19:30:04.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.397+0000 7fa5b17fa640 10 client.4558.objecter ms_handle_connect 0x7fa5a806df70 2024-03-25T19:30:04.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.401+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fa5b8054880 con 0x7fa5bc0b1730 2024-03-25T19:30:04.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.401+0000 7fa5b17fa640 10 client.4558.objecter ms_dispatch 0x7fa5bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:04.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.401+0000 7fa5b17fa640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:04.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.401+0000 7fa5b17fa640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:04.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.401+0000 7fa5b17fa640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T19:30:04.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.401+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5b8050390 con 0x7fa5bc0b1730 2024-03-25T19:30:04.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.401+0000 7fa5b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:04.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.401+0000 7fa5b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:04.657 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.656+0000 7fa5c2f82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.657 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.656+0000 7fa5c2f82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.657 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.656+0000 7fa5c2f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:04.658 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.656+0000 7fa5c2f82640 1 -- 172.21.15.123:0/1051996244 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5bc0a94c0 con 0x7fa5bc0b1730 2024-03-25T19:30:04.680 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.679+0000 7fa5b17fa640 1 -- 172.21.15.123:0/1051996244 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fa5bc0a94c0 con 0x7fa5bc0b1730 2024-03-25T19:30:04.680 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.679+0000 7fa5b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.680 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.679+0000 7fa5b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:04.681 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:30:04.681 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:04.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 10 librados: watch_flush enter 2024-03-25T19:30:04.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 10 librados: watch_flush exit 2024-03-25T19:30:04.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T19:30:04.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 10 client.4558.objecter successfully canceled tick 2024-03-25T19:30:04.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 1 -- 172.21.15.123:0/1051996244 >> v1:172.21.15.151:6800/727684239 conn(0x7fa5a806df70 legacy=0x7fa5a8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 10 monclient: shutdown 2024-03-25T19:30:04.683 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:04.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 1 -- 172.21.15.123:0/1051996244 >> v1:172.21.15.151:6789/0 conn(0x7fa5bc0b1730 legacy=0x7fa5bc1d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 1 -- 172.21.15.123:0/1051996244 shutdown_connections 2024-03-25T19:30:04.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 1 -- 172.21.15.123:0/1051996244 >> 172.21.15.123:0/1051996244 conn(0x7fa5bc0617a0 msgr2=0x7fa5bc0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:04.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 1 -- 172.21.15.123:0/1051996244 shutdown_connections 2024-03-25T19:30:04.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 1 -- 172.21.15.123:0/1051996244 wait complete. 2024-03-25T19:30:04.684 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:04.682+0000 7fa5c2f82640 1 librados: shutdown 2024-03-25T19:30:04.694 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:11.500 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:11.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.614+0000 7f22cbcf4640 10 monclient: get_monmap_and_config 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.614+0000 7f22cbcf4640 10 monclient: build_initial_monmap 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.615+0000 7f22cbcf4640 10 monclient: monmap: 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:30:11.615889+0000 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:30:11.615889+0000 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:30:11.615 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.615+0000 7f22cbcf4640 1 Processor -- start 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.615+0000 7f22cbcf4640 1 -- start start 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.615+0000 7f22cbcf4640 10 monclient: init 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.616+0000 7f22cbcf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.616+0000 7f22cbcf4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.616+0000 7f22cbcf4640 10 monclient(hunting): picked mon.noname-b con 0x7f22c40b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.616+0000 7f22cbcf4640 10 monclient(hunting): picked mon.noname-a con 0x7f22c40b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.616+0000 7f22cbcf4640 10 monclient(hunting): picked mon.noname-c con 0x7f22c414be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.616+0000 7f22cbcf4640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c40af940 con 0x7f22c40b1730 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.616+0000 7f22cbcf4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c40b0a70 con 0x7f22c414be90 2024-03-25T19:30:11.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.616+0000 7f22cbcf4640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c40b0d50 con 0x7f22c40b4d60 2024-03-25T19:30:11.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.616+0000 7f22cbcf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:11.618 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.616+0000 7f22cbcf4640 10 monclient(hunting): authenticate will time out at 6048.976398s 2024-03-25T19:30:11.622 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.623+0000 7f22c9a90640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f22c40b4d60 0x7f22c413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:43528/0 (socket says 172.21.15.123:43528) 2024-03-25T19:30:11.623 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.623+0000 7f22c9a90640 1 -- 172.21.15.123:0/2726837531 learned_addr learned my addr 172.21.15.123:0/2726837531 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:11.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.629+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577703630 0 0) 0x7f22c40b0d50 con 0x7f22c40b4d60 2024-03-25T19:30:11.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.629+0000 7f22c8a8e640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.629+0000 7f22c8a8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.629+0000 7f22c8a8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.629+0000 7f22c8a8e640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:30:11.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.629+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b0003520 con 0x7f22c40b4d60 2024-03-25T19:30:11.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.633+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396333268 0 0) 0x7f22c40b0a70 con 0x7f22c414be90 2024-03-25T19:30:11.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.633+0000 7f22c8a8e640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.633+0000 7f22c8a8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.633+0000 7f22c8a8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.633+0000 7f22c8a8e640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:30:11.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.633+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c40b0d50 con 0x7f22c414be90 2024-03-25T19:30:11.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3559816866 0 0) 0x7f22b0003520 con 0x7f22c40b4d60 2024-03-25T19:30:11.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 >> v1:172.21.15.151:6789/0 conn(0x7f22c414be90 legacy=0x7f22c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 >> v1:172.21.15.123:6789/0 conn(0x7f22c40b1730 legacy=0x7f22c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 10 monclient: _finish_hunting 0 2024-03-25T19:30:11.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 1 monclient: found mon.noname-b 2024-03-25T19:30:11.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:11.637 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c40b1030 con 0x7f22c40b4d60 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 10 monclient: _finish_auth 0 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 10 monclient: _check_auth_tickets 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f22c0001690 con 0x7f22c40b4d60 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22c8a8e640 10 monclient: dump: 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:11.638 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:11.639 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:11.639 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:11.639 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22cbcf4640 5 monclient: authenticate success, global_id 4583 2024-03-25T19:30:11.639 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.637+0000 7f22cbcf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:11.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.642+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22c00017f0 con 0x7f22c40b4d60 2024-03-25T19:30:11.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.642+0000 7f22c8a8e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:11.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.642+0000 7f22c8a8e640 1 -- 172.21.15.123:0/2726837531 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f22c0001c00 con 0x7f22c40b4d60 2024-03-25T19:30:11.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.642+0000 7f22cbcf4640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:11.642 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.642+0000 7f22cbcf4640 4 set_mon_vals no callback set 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.642+0000 7f22c8a8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.642+0000 7f22c8a8e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.642+0000 7f22c8a8e640 10 monclient: dump: 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:11.643 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:11.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.643+0000 7f22cbcf4640 10 monclient: shutdown 2024-03-25T19:30:11.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.643+0000 7f22cbcf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:11.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.643+0000 7f22cbcf4640 1 -- 172.21.15.123:0/2726837531 >> v1:172.21.15.123:6790/0 conn(0x7f22c40b4d60 legacy=0x7f22c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.643+0000 7f22cbcf4640 1 -- 172.21.15.123:0/2726837531 shutdown_connections 2024-03-25T19:30:11.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.643+0000 7f22cbcf4640 1 -- 172.21.15.123:0/2726837531 >> 172.21.15.123:0/2726837531 conn(0x7f22c40617a0 msgr2=0x7f22c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:11.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.643+0000 7f22cbcf4640 1 -- 172.21.15.123:0/2726837531 shutdown_connections 2024-03-25T19:30:11.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.643+0000 7f22cbcf4640 1 -- 172.21.15.123:0/2726837531 wait complete. 2024-03-25T19:30:11.645 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.644+0000 7f22cbcf4640 10 monclient: build_initial_monmap 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.644+0000 7f22cbcf4640 10 monclient: monmap: 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.644+0000 7f22cbcf4640 1 librados: starting msgr at 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.644+0000 7f22cbcf4640 1 librados: starting objecter 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.644+0000 7f22cbcf4640 1 Processor -- start 2024-03-25T19:30:11.646 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.645+0000 7f22cbcf4640 1 -- start start 2024-03-25T19:30:11.647 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.645+0000 7f22cbcf4640 1 librados: setting wanted keys 2024-03-25T19:30:11.647 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.645+0000 7f22cbcf4640 1 librados: calling monclient init 2024-03-25T19:30:11.647 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.645+0000 7f22cbcf4640 10 monclient: init 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.645+0000 7f22cbcf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.645+0000 7f22cbcf4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.645+0000 7f22cbcf4640 10 monclient(hunting): picked mon.noname-a con 0x7f22c40b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.645+0000 7f22cbcf4640 10 monclient(hunting): picked mon.noname-c con 0x7f22c40b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.645+0000 7f22cbcf4640 10 monclient(hunting): picked mon.noname-b con 0x7f22c414be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.646+0000 7f22cbcf4640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c4135e10 con 0x7f22c40b1730 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.646+0000 7f22cbcf4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c4143900 con 0x7f22c414be90 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.646+0000 7f22cbcf4640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c412f8b0 con 0x7f22c40b4d60 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.646+0000 7f22cbcf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.646+0000 7f22cbcf4640 10 monclient(hunting): authenticate will time out at 6049.005976s 2024-03-25T19:30:11.649 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.649+0000 7f22ca291640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f22c414be90 0x7f22c412f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:56052/0 (socket says 172.21.15.123:56052) 2024-03-25T19:30:11.649 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.649+0000 7f22ca291640 1 -- 172.21.15.123:0/4118761838 learned_addr learned my addr 172.21.15.123:0/4118761838 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:11.650 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.650+0000 7f22ba7fc640 10 client.?.objecter ms_handle_connect 0x7f22c414be90 2024-03-25T19:30:11.650 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.650+0000 7f22ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:11.653 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.654+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724416614 0 0) 0x7f22c4143900 con 0x7f22c414be90 2024-03-25T19:30:11.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.654+0000 7f22ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.654+0000 7f22ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.654+0000 7f22ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.654+0000 7f22ba7fc640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:30:11.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.654+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a8004eb0 con 0x7f22c414be90 2024-03-25T19:30:11.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.654+0000 7f22ba7fc640 10 client.?.objecter ms_handle_connect 0x7f22c40b1730 2024-03-25T19:30:11.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.654+0000 7f22ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:11.664 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.664+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449710436 0 0) 0x7f22c4135e10 con 0x7f22c40b1730 2024-03-25T19:30:11.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.664+0000 7f22ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.664+0000 7f22ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.664+0000 7f22ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.664+0000 7f22ba7fc640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:30:11.665 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.664+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c4143900 con 0x7f22c40b1730 2024-03-25T19:30:11.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.671+0000 7f22ba7fc640 10 client.?.objecter ms_handle_connect 0x7f22c40b4d60 2024-03-25T19:30:11.671 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.671+0000 7f22ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.674+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3390129885 0 0) 0x7f22a8004eb0 con 0x7f22c414be90 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.674+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22c4135e10 con 0x7f22c414be90 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.674+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f22bc001c70 con 0x7f22c414be90 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.674+0000 7f22ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.674+0000 7f22ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.674+0000 7f22ba7fc640 10 monclient(hunting): dump: 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:11.674 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:11.675 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:11.675 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:11.675 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:11.675 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.674+0000 7f22ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:11.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.684+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2294104168 0 0) 0x7f22c4135e10 con 0x7f22c414be90 2024-03-25T19:30:11.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.684+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 >> v1:172.21.15.123:6790/0 conn(0x7f22c40b4d60 legacy=0x7f22c412ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.684+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 >> v1:172.21.15.123:6789/0 conn(0x7f22c40b1730 legacy=0x7f22c41d8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.684+0000 7f22ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:11.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.684+0000 7f22ba7fc640 1 monclient: found mon.b 2024-03-25T19:30:11.685 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.684+0000 7f22ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.684+0000 7f22ba7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.684+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c41371c0 con 0x7f22c414be90 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.684+0000 7f22ba7fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22cbcf4640 5 monclient: authenticate success, global_id 4564 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22cbcf4640 10 monclient: _renew_subs 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22cbcf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22cbcf4640 1 -- 172.21.15.123:0/4118761838 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c4138890 con 0x7f22c414be90 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22cbcf4640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22cbcf4640 10 monclient: _renew_subs 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22cbcf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22cbcf4640 1 -- 172.21.15.123:0/4118761838 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c41361c0 con 0x7f22c414be90 2024-03-25T19:30:11.686 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.685+0000 7f22cbcf4640 1 librados: init done 2024-03-25T19:30:11.687 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.686+0000 7f22cbcf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:11.687 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.686+0000 7f22cbcf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:11.687 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.686+0000 7f22cbcf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:11.687 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.686+0000 7f22cbcf4640 1 -- 172.21.15.123:0/4118761838 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c41373b0 con 0x7f22c414be90 2024-03-25T19:30:11.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.957+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22bc003970 con 0x7f22c414be90 2024-03-25T19:30:11.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.957+0000 7f22ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:11.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.957+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f22bc003d40 con 0x7f22c414be90 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.957+0000 7f22ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.957+0000 7f22ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.957+0000 7f22ba7fc640 10 monclient: dump: 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:11.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.957+0000 7f22bbfff640 4 set_mon_vals no callback set 2024-03-25T19:30:11.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.958+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f22bc003f60 con 0x7f22c414be90 2024-03-25T19:30:11.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.959+0000 7f22c9a90640 10 monclient: get_auth_request con 0x7f22a806dd10 auth_method 0 2024-03-25T19:30:11.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.959+0000 7f22ba7fc640 10 client.4564.objecter ms_handle_connect 0x7f22a806dd10 2024-03-25T19:30:11.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.962+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f22bc054a00 con 0x7f22c414be90 2024-03-25T19:30:11.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.962+0000 7f22ba7fc640 10 client.4564.objecter ms_dispatch 0x7f22c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:11.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.962+0000 7f22ba7fc640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:11.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.962+0000 7f22ba7fc640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:11.962 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.962+0000 7f22ba7fc640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T19:30:11.963 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.964+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22bc050510 con 0x7f22c414be90 2024-03-25T19:30:11.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.964+0000 7f22ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:11.964 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:11.964+0000 7f22ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:12.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.184+0000 7f22cbcf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:12.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.184+0000 7f22cbcf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:12.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.184+0000 7f22cbcf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:12.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.184+0000 7f22cbcf4640 1 -- 172.21.15.123:0/4118761838 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22c41d8520 con 0x7f22c414be90 2024-03-25T19:30:12.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.199+0000 7f22ba7fc640 1 -- 172.21.15.123:0/4118761838 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f22c41d8520 con 0x7f22c414be90 2024-03-25T19:30:12.200 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.200+0000 7f22ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:12.200 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.200+0000 7f22ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:12.200 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:30:12.200 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:12.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 10 librados: watch_flush enter 2024-03-25T19:30:12.202 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 10 librados: watch_flush exit 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 10 client.4564.objecter successfully canceled tick 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 1 -- 172.21.15.123:0/4118761838 >> v1:172.21.15.151:6800/727684239 conn(0x7f22a806dd10 legacy=0x7f22a80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 10 monclient: shutdown 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 1 -- 172.21.15.123:0/4118761838 >> v1:172.21.15.151:6789/0 conn(0x7f22c414be90 legacy=0x7f22c412f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 1 -- 172.21.15.123:0/4118761838 shutdown_connections 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 1 -- 172.21.15.123:0/4118761838 >> 172.21.15.123:0/4118761838 conn(0x7f22c40617a0 msgr2=0x7f22c40b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 1 -- 172.21.15.123:0/4118761838 shutdown_connections 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 1 -- 172.21.15.123:0/4118761838 wait complete. 2024-03-25T19:30:12.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:12.202+0000 7f22cbcf4640 1 librados: shutdown 2024-03-25T19:30:12.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:16.132 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:08:05.704 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:16.182 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:08:05.754 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:16.218 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:08:05.790 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:16.233 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:08:05.805 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:16.238 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:08:05.771 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:16.249 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:08:05.782 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:16.251 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:08:05.823 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:16.277 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:08:05.849 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:16.281 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:08:05.814 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:19.022 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:19.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.135+0000 7f4b6ae64640 10 monclient: get_monmap_and_config 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.135+0000 7f4b6ae64640 10 monclient: build_initial_monmap 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.136+0000 7f4b6ae64640 10 monclient: monmap: 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:30:19.136906+0000 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:30:19.136906+0000 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:30:19.136 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.136+0000 7f4b6ae64640 1 Processor -- start 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 1 -- start start 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 10 monclient: init 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 10 monclient(hunting): picked mon.noname-a con 0x7f4b640b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 10 monclient(hunting): picked mon.noname-b con 0x7f4b640b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 10 monclient(hunting): picked mon.noname-c con 0x7f4b6414be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b640af940 con 0x7f4b640b4d60 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b640b0a70 con 0x7f4b6414be90 2024-03-25T19:30:19.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b640b0d50 con 0x7f4b640b1730 2024-03-25T19:30:19.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:19.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.137+0000 7f4b6ae64640 10 monclient(hunting): authenticate will time out at 6056.497619s 2024-03-25T19:30:19.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.147+0000 7f4b69401640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4b6414be90 0x7f4b6414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:41004/0 (socket says 172.21.15.123:41004) 2024-03-25T19:30:19.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.148+0000 7f4b69401640 1 -- 172.21.15.123:0/1684685762 learned_addr learned my addr 172.21.15.123:0/1684685762 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:19.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.155+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98051213 0 0) 0x7f4b640b0a70 con 0x7f4b6414be90 2024-03-25T19:30:19.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.155+0000 7f4b637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.155+0000 7f4b637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.155+0000 7f4b637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.155+0000 7f4b637fe640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:30:19.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.155+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b54003520 con 0x7f4b6414be90 2024-03-25T19:30:19.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.162+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12091666 0 0) 0x7f4b640af940 con 0x7f4b640b4d60 2024-03-25T19:30:19.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.162+0000 7f4b637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.162+0000 7f4b637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.162+0000 7f4b637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.162+0000 7f4b637fe640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:30:19.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.162+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b640b0a70 con 0x7f4b640b4d60 2024-03-25T19:30:19.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 202308045 0 0) 0x7f4b54003520 con 0x7f4b6414be90 2024-03-25T19:30:19.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 >> v1:172.21.15.123:6790/0 conn(0x7f4b640b1730 legacy=0x7f4b640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 >> v1:172.21.15.123:6789/0 conn(0x7f4b640b4d60 legacy=0x7f4b6413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:19.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 1 monclient: found mon.noname-c 2024-03-25T19:30:19.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:19.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b640b1030 con 0x7f4b6414be90 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4b5c001a10 con 0x7f4b6414be90 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b637fe640 10 monclient: dump: 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.166 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:19.167 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:19.167 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:19.167 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:19.167 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:19.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b6ae64640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:30:19.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.165+0000 7f4b6ae64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:19.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.181+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b5c001b70 con 0x7f4b6414be90 2024-03-25T19:30:19.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.181+0000 7f4b637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:19.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.181+0000 7f4b637fe640 1 -- 172.21.15.123:0/1684685762 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4b5c001f80 con 0x7f4b6414be90 2024-03-25T19:30:19.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.181+0000 7f4b637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.181+0000 7f4b637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:19.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.181+0000 7f4b637fe640 10 monclient: dump: 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.181+0000 7f4b6ae64640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:19.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.181+0000 7f4b6ae64640 4 set_mon_vals no callback set 2024-03-25T19:30:19.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.182+0000 7f4b6ae64640 10 monclient: shutdown 2024-03-25T19:30:19.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.182+0000 7f4b6ae64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:19.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.182+0000 7f4b6ae64640 1 -- 172.21.15.123:0/1684685762 >> v1:172.21.15.151:6789/0 conn(0x7f4b6414be90 legacy=0x7f4b6414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.182+0000 7f4b6ae64640 1 -- 172.21.15.123:0/1684685762 shutdown_connections 2024-03-25T19:30:19.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.182+0000 7f4b6ae64640 1 -- 172.21.15.123:0/1684685762 >> 172.21.15.123:0/1684685762 conn(0x7f4b640617a0 msgr2=0x7f4b64061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:19.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.182+0000 7f4b6ae64640 1 -- 172.21.15.123:0/1684685762 shutdown_connections 2024-03-25T19:30:19.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.182+0000 7f4b6ae64640 1 -- 172.21.15.123:0/1684685762 wait complete. 2024-03-25T19:30:19.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.183+0000 7f4b6ae64640 10 monclient: build_initial_monmap 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.183+0000 7f4b6ae64640 10 monclient: monmap: 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.184+0000 7f4b6ae64640 1 librados: starting msgr at 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.184+0000 7f4b6ae64640 1 librados: starting objecter 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.184+0000 7f4b6ae64640 1 Processor -- start 2024-03-25T19:30:19.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.184+0000 7f4b6ae64640 1 -- start start 2024-03-25T19:30:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.184+0000 7f4b6ae64640 1 librados: setting wanted keys 2024-03-25T19:30:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.184+0000 7f4b6ae64640 1 librados: calling monclient init 2024-03-25T19:30:19.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.184+0000 7f4b6ae64640 10 monclient: init 2024-03-25T19:30:19.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.185+0000 7f4b6ae64640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:19.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.185+0000 7f4b6ae64640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:19.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.185+0000 7f4b6ae64640 10 monclient(hunting): picked mon.noname-b con 0x7f4b640b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:19.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.185+0000 7f4b6ae64640 10 monclient(hunting): picked mon.noname-c con 0x7f4b640b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:19.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.185+0000 7f4b6ae64640 10 monclient(hunting): picked mon.noname-a con 0x7f4b6414be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:19.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.185+0000 7f4b6ae64640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b64144c40 con 0x7f4b6414be90 2024-03-25T19:30:19.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.185+0000 7f4b6ae64640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b64137ef0 con 0x7f4b640b1730 2024-03-25T19:30:19.188 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.185+0000 7f4b6ae64640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6413b270 con 0x7f4b640b4d60 2024-03-25T19:30:19.188 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.185+0000 7f4b6ae64640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:19.188 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.185+0000 7f4b6ae64640 10 monclient(hunting): authenticate will time out at 6056.545182s 2024-03-25T19:30:19.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.196+0000 7f4b63fff640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f4b640b4d60 0x7f4b64136be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:47348/0 (socket says 172.21.15.123:47348) 2024-03-25T19:30:19.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.196+0000 7f4b63fff640 1 -- 172.21.15.123:0/987333813 learned_addr learned my addr 172.21.15.123:0/987333813 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:19.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.198+0000 7f4b617fa640 10 client.?.objecter ms_handle_connect 0x7f4b640b4d60 2024-03-25T19:30:19.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.198+0000 7f4b617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.199+0000 7f4b617fa640 10 client.?.objecter ms_handle_connect 0x7f4b6414be90 2024-03-25T19:30:19.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.199+0000 7f4b617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:19.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.208+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755926931 0 0) 0x7f4b6413b270 con 0x7f4b640b4d60 2024-03-25T19:30:19.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.208+0000 7f4b617fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.208+0000 7f4b617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.208+0000 7f4b617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.208+0000 7f4b617fa640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:30:19.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.208+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b48004eb0 con 0x7f4b640b4d60 2024-03-25T19:30:19.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.216+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448185473 0 0) 0x7f4b64144c40 con 0x7f4b6414be90 2024-03-25T19:30:19.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.216+0000 7f4b617fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.216+0000 7f4b617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.216+0000 7f4b617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.216+0000 7f4b617fa640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:30:19.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.216+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b6413b270 con 0x7f4b6414be90 2024-03-25T19:30:19.219 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.219+0000 7f4b617fa640 10 client.?.objecter ms_handle_connect 0x7f4b640b1730 2024-03-25T19:30:19.219 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.219+0000 7f4b617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:19.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.221+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3282684905 0 0) 0x7f4b48004eb0 con 0x7f4b640b4d60 2024-03-25T19:30:19.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.221+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b64144c40 con 0x7f4b640b4d60 2024-03-25T19:30:19.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.221+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4b50001a30 con 0x7f4b640b4d60 2024-03-25T19:30:19.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.221+0000 7f4b617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.221+0000 7f4b617fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.221+0000 7f4b617fa640 10 monclient(hunting): dump: 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:19.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.221+0000 7f4b617fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:19.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.225+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216291107 0 0) 0x7f4b64137ef0 con 0x7f4b640b1730 2024-03-25T19:30:19.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.225+0000 7f4b617fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.225+0000 7f4b617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.225+0000 7f4b617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.226 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.225+0000 7f4b617fa640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:30:19.226 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.225+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b48004eb0 con 0x7f4b640b1730 2024-03-25T19:30:19.237 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1833009057 0 0) 0x7f4b64144c40 con 0x7f4b640b4d60 2024-03-25T19:30:19.237 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 >> v1:172.21.15.151:6789/0 conn(0x7f4b640b1730 legacy=0x7f4b6413b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.237 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 >> v1:172.21.15.123:6789/0 conn(0x7f4b6414be90 legacy=0x7f4b641e70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 10 monclient: _finish_hunting 0 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 1 monclient: found mon.c 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b64144910 con 0x7f4b640b4d60 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 10 monclient: _finish_auth 0 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 10 monclient: _check_auth_tickets 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b6ae64640 5 monclient: authenticate success, global_id 4592 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b6ae64640 10 monclient: _renew_subs 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b6ae64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b6ae64640 1 -- 172.21.15.123:0/987333813 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b64138140 con 0x7f4b640b4d60 2024-03-25T19:30:19.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b6ae64640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b6ae64640 10 monclient: _renew_subs 2024-03-25T19:30:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b6ae64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b6ae64640 1 -- 172.21.15.123:0/987333813 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b641394b0 con 0x7f4b640b4d60 2024-03-25T19:30:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.237+0000 7f4b6ae64640 1 librados: init done 2024-03-25T19:30:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.238+0000 7f4b6ae64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.238+0000 7f4b6ae64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.238+0000 7f4b6ae64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:19.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.238+0000 7f4b6ae64640 1 -- 172.21.15.123:0/987333813 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b640ae6c0 con 0x7f4b640b4d60 2024-03-25T19:30:19.452 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.452+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b50001c70 con 0x7f4b640b4d60 2024-03-25T19:30:19.452 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.452+0000 7f4b617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:19.452 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.452+0000 7f4b62ffd640 4 set_mon_vals no callback set 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.519+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4b50003a60 con 0x7f4b640b4d60 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.519+0000 7f4b617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.519+0000 7f4b617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.519+0000 7f4b617fa640 10 monclient: dump: 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:19.519 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:19.520 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:19.520 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:19.520 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:19.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.520+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f4b50003c40 con 0x7f4b640b4d60 2024-03-25T19:30:19.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.521+0000 7f4b68c00640 10 monclient: get_auth_request con 0x7f4b4806de00 auth_method 0 2024-03-25T19:30:19.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.521+0000 7f4b617fa640 10 client.4592.objecter ms_handle_connect 0x7f4b4806de00 2024-03-25T19:30:19.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.526+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f4b50054950 con 0x7f4b640b4d60 2024-03-25T19:30:19.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.526+0000 7f4b617fa640 10 client.4592.objecter ms_dispatch 0x7f4b64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:19.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.526+0000 7f4b617fa640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:19.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.526+0000 7f4b617fa640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:19.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.526+0000 7f4b617fa640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-25T19:30:19.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.526+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b50050290 con 0x7f4b640b4d60 2024-03-25T19:30:19.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.526+0000 7f4b617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:19.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.526+0000 7f4b617fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.776+0000 7f4b6ae64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.776+0000 7f4b6ae64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.776+0000 7f4b6ae64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.776+0000 7f4b6ae64640 1 -- 172.21.15.123:0/987333813 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b64144df0 con 0x7f4b640b4d60 2024-03-25T19:30:19.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.821+0000 7f4b617fa640 1 -- 172.21.15.123:0/987333813 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f4b64144df0 con 0x7f4b640b4d60 2024-03-25T19:30:19.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.821+0000 7f4b617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:19.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.821+0000 7f4b617fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:19.821 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:30:19.821 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:19.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 10 librados: watch_flush enter 2024-03-25T19:30:19.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 10 librados: watch_flush exit 2024-03-25T19:30:19.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-25T19:30:19.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 10 client.4592.objecter successfully canceled tick 2024-03-25T19:30:19.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 1 -- 172.21.15.123:0/987333813 >> v1:172.21.15.151:6800/727684239 conn(0x7f4b4806de00 legacy=0x7f4b480702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 10 monclient: shutdown 2024-03-25T19:30:19.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:19.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 1 -- 172.21.15.123:0/987333813 >> v1:172.21.15.123:6790/0 conn(0x7f4b640b4d60 legacy=0x7f4b64136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 1 -- 172.21.15.123:0/987333813 shutdown_connections 2024-03-25T19:30:19.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 1 -- 172.21.15.123:0/987333813 >> 172.21.15.123:0/987333813 conn(0x7f4b640617a0 msgr2=0x7f4b6413c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:19.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 1 -- 172.21.15.123:0/987333813 shutdown_connections 2024-03-25T19:30:19.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 1 -- 172.21.15.123:0/987333813 wait complete. 2024-03-25T19:30:19.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:19.823+0000 7f4b6ae64640 1 librados: shutdown 2024-03-25T19:30:19.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:26.639 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:26.748 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.748+0000 7fca33725640 10 monclient: get_monmap_and_config 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.748+0000 7fca33725640 10 monclient: build_initial_monmap 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.748+0000 7fca33725640 10 monclient: monmap: 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:30:26.749700+0000 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:30:26.749700+0000 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:30:26.749 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.749+0000 7fca33725640 1 Processor -- start 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.749+0000 7fca33725640 1 -- start start 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.749+0000 7fca33725640 10 monclient: init 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca33725640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca33725640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca33725640 10 monclient(hunting): picked mon.noname-c con 0x7fca2c0b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca33725640 10 monclient(hunting): picked mon.noname-a con 0x7fca2c0b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca33725640 10 monclient(hunting): picked mon.noname-b con 0x7fca2c14be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca33725640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c0af940 con 0x7fca2c0b1730 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca33725640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c0b0a70 con 0x7fca2c0b4d60 2024-03-25T19:30:26.751 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca33725640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c0b0d50 con 0x7fca2c14be90 2024-03-25T19:30:26.752 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca33725640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:26.752 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca33725640 10 monclient(hunting): authenticate will time out at 6064.110134s 2024-03-25T19:30:26.752 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca314c1640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fca2c0b4d60 0x7fca2c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:51694/0 (socket says 172.21.15.123:51694) 2024-03-25T19:30:26.752 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.750+0000 7fca314c1640 1 -- 172.21.15.123:0/629324115 learned_addr learned my addr 172.21.15.123:0/629324115 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:26.803 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.804+0000 7fca23fff640 1 -- 172.21.15.123:0/629324115 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064107560 0 0) 0x7fca2c0af940 con 0x7fca2c0b1730 2024-03-25T19:30:26.804 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.804+0000 7fca23fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.804 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.804+0000 7fca23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.804 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.804+0000 7fca23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.804 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.804+0000 7fca23fff640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:30:26.804 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.804+0000 7fca23fff640 1 -- 172.21.15.123:0/629324115 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca18003520 con 0x7fca2c0b1730 2024-03-25T19:30:26.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 1 -- 172.21.15.123:0/629324115 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1573130744 0 0) 0x7fca18003520 con 0x7fca2c0b1730 2024-03-25T19:30:26.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 1 -- 172.21.15.123:0/629324115 >> v1:172.21.15.123:6790/0 conn(0x7fca2c14be90 legacy=0x7fca2c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 1 -- 172.21.15.123:0/629324115 >> v1:172.21.15.151:6789/0 conn(0x7fca2c0b4d60 legacy=0x7fca2c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:26.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 1 monclient: found mon.noname-a 2024-03-25T19:30:26.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:26.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 1 -- 172.21.15.123:0/629324115 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c0b1030 con 0x7fca2c0b1730 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 10 monclient: _finish_auth 0 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 1 -- 172.21.15.123:0/629324115 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fca1c001690 con 0x7fca2c0b1730 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca33725640 5 monclient: authenticate success, global_id 4560 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca33725640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca23fff640 10 monclient: dump: 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:26.820 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.821 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:26.821 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:26.821 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:26.821 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:26.821 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:26.821 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.819+0000 7fca33725640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:26.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.825+0000 7fca23fff640 1 -- 172.21.15.123:0/629324115 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca1c0017f0 con 0x7fca2c0b1730 2024-03-25T19:30:26.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.825+0000 7fca23fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:26.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.825+0000 7fca23fff640 1 -- 172.21.15.123:0/629324115 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fca1c001c00 con 0x7fca2c0b1730 2024-03-25T19:30:26.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.825+0000 7fca23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.825+0000 7fca23fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:26.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.825+0000 7fca23fff640 10 monclient: dump: 2024-03-25T19:30:26.826 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:26.826 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.825+0000 7fca33725640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.825+0000 7fca33725640 4 set_mon_vals no callback set 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.826+0000 7fca33725640 10 monclient: shutdown 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.826+0000 7fca33725640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.826+0000 7fca33725640 1 -- 172.21.15.123:0/629324115 >> v1:172.21.15.123:6789/0 conn(0x7fca2c0b1730 legacy=0x7fca2c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.826+0000 7fca33725640 1 -- 172.21.15.123:0/629324115 shutdown_connections 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.826+0000 7fca33725640 1 -- 172.21.15.123:0/629324115 >> 172.21.15.123:0/629324115 conn(0x7fca2c0617a0 msgr2=0x7fca2c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:26.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.826+0000 7fca33725640 1 -- 172.21.15.123:0/629324115 shutdown_connections 2024-03-25T19:30:26.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.826+0000 7fca33725640 1 -- 172.21.15.123:0/629324115 wait complete. 2024-03-25T19:30:26.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.827+0000 7fca33725640 10 monclient: build_initial_monmap 2024-03-25T19:30:26.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.827+0000 7fca33725640 10 monclient: monmap: 2024-03-25T19:30:26.829 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:26.829 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:26.829 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:30:26.829 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:30:26.829 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:26.829 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:30:26.829 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.827+0000 7fca33725640 1 librados: starting msgr at 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.827+0000 7fca33725640 1 librados: starting objecter 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.827+0000 7fca33725640 1 Processor -- start 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.827+0000 7fca33725640 1 -- start start 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.827+0000 7fca33725640 1 librados: setting wanted keys 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.827+0000 7fca33725640 1 librados: calling monclient init 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.827+0000 7fca33725640 10 monclient: init 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.828+0000 7fca33725640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.828+0000 7fca33725640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.828+0000 7fca33725640 10 monclient(hunting): picked mon.noname-b con 0x7fca2c0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.828+0000 7fca33725640 10 monclient(hunting): picked mon.noname-a con 0x7fca2c0b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:26.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.828+0000 7fca33725640 10 monclient(hunting): picked mon.noname-c con 0x7fca2c14be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:26.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.828+0000 7fca33725640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c1d7940 con 0x7fca2c0b4d60 2024-03-25T19:30:26.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.828+0000 7fca33725640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c1d8570 con 0x7fca2c0b1730 2024-03-25T19:30:26.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.828+0000 7fca33725640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c1d8700 con 0x7fca2c14be90 2024-03-25T19:30:26.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.828+0000 7fca33725640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:26.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.828+0000 7fca33725640 10 monclient(hunting): authenticate will time out at 6064.188589s 2024-03-25T19:30:26.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.837+0000 7fca314c1640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fca2c0b1730 0x7fca2c1d7e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:51738/0 (socket says 172.21.15.123:51738) 2024-03-25T19:30:26.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.838+0000 7fca314c1640 1 -- 172.21.15.123:0/4035618367 learned_addr learned my addr 172.21.15.123:0/4035618367 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:26.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.839+0000 7fca21ffb640 10 client.?.objecter ms_handle_connect 0x7fca2c0b1730 2024-03-25T19:30:26.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.839+0000 7fca21ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:26.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.844+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002180711 0 0) 0x7fca2c1d8570 con 0x7fca2c0b1730 2024-03-25T19:30:26.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.844+0000 7fca21ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.844+0000 7fca21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.844+0000 7fca21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.844+0000 7fca21ffb640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:30:26.845 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.844+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca10004eb0 con 0x7fca2c0b1730 2024-03-25T19:30:26.848 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.848+0000 7fca21ffb640 10 client.?.objecter ms_handle_connect 0x7fca2c0b4d60 2024-03-25T19:30:26.848 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.848+0000 7fca21ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:26.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.856+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102466047 0 0) 0x7fca10004eb0 con 0x7fca2c0b1730 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.857+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca2c1d8570 con 0x7fca2c0b1730 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.857+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fca24001c30 con 0x7fca2c0b1730 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.857+0000 7fca21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.857+0000 7fca21ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.857+0000 7fca21ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:26.857 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:26.858 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:26.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.857+0000 7fca21ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:26.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.863+0000 7fca21ffb640 10 client.?.objecter ms_handle_connect 0x7fca2c14be90 2024-03-25T19:30:26.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.863+0000 7fca21ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:26.867 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 402119643 0 0) 0x7fca2c1d8570 con 0x7fca2c0b1730 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 >> v1:172.21.15.123:6790/0 conn(0x7fca2c14be90 legacy=0x7fca2c12fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 >> v1:172.21.15.123:6789/0 conn(0x7fca2c0b4d60 legacy=0x7fca2c136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 1 monclient: found mon.b 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c1361c0 con 0x7fca2c0b1730 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca33725640 5 monclient: authenticate success, global_id 4576 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca33725640 10 monclient: _renew_subs 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca33725640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:26.868 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.867+0000 7fca33725640 1 -- 172.21.15.123:0/4035618367 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c136470 con 0x7fca2c0b1730 2024-03-25T19:30:26.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.868+0000 7fca33725640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:26.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.868+0000 7fca33725640 10 monclient: _renew_subs 2024-03-25T19:30:26.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.868+0000 7fca33725640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:26.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.868+0000 7fca33725640 1 -- 172.21.15.123:0/4035618367 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c137790 con 0x7fca2c0b1730 2024-03-25T19:30:26.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.868+0000 7fca33725640 1 librados: init done 2024-03-25T19:30:26.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.869+0000 7fca33725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:26.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.869+0000 7fca33725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:26.869 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.869+0000 7fca33725640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:26.870 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:26.869+0000 7fca33725640 1 -- 172.21.15.123:0/4035618367 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c0627b0 con 0x7fca2c0b1730 2024-03-25T19:30:27.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.139+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca24004b60 con 0x7fca2c0b1730 2024-03-25T19:30:27.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.139+0000 7fca21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:27.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.139+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fca24004f70 con 0x7fca2c0b1730 2024-03-25T19:30:27.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.139+0000 7fca21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.139+0000 7fca21ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.139+0000 7fca21ffb640 10 monclient: dump: 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:27.140 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.139+0000 7fca237fe640 4 set_mon_vals no callback set 2024-03-25T19:30:27.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.140+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fca24005150 con 0x7fca2c0b1730 2024-03-25T19:30:27.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.141+0000 7fca30cc0640 10 monclient: get_auth_request con 0x7fca1006dc00 auth_method 0 2024-03-25T19:30:27.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.141+0000 7fca21ffb640 10 client.4576.objecter ms_handle_connect 0x7fca1006dc00 2024-03-25T19:30:27.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.148+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fca24054c60 con 0x7fca2c0b1730 2024-03-25T19:30:27.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.148+0000 7fca21ffb640 10 client.4576.objecter ms_dispatch 0x7fca2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:27.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.148+0000 7fca21ffb640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:27.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.148+0000 7fca21ffb640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:27.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.148+0000 7fca21ffb640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T19:30:27.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.149+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca24050770 con 0x7fca2c0b1730 2024-03-25T19:30:27.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.149+0000 7fca21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:27.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.149+0000 7fca21ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:27.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.409+0000 7fca33725640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:27.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.409+0000 7fca33725640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:27.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.409+0000 7fca33725640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:30:27.409 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.409+0000 7fca33725640 1 -- 172.21.15.123:0/4035618367 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca2c069f40 con 0x7fca2c0b1730 2024-03-25T19:30:27.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.435+0000 7fca21ffb640 1 -- 172.21.15.123:0/4035618367 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fca240526a0 con 0x7fca2c0b1730 2024-03-25T19:30:27.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.435+0000 7fca21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:27.435 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.435+0000 7fca21ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:27.435 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:30:27.435 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 10 librados: watch_flush enter 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 10 librados: watch_flush exit 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 10 client.4576.objecter successfully canceled tick 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 1 -- 172.21.15.123:0/4035618367 >> v1:172.21.15.151:6800/727684239 conn(0x7fca1006dc00 legacy=0x7fca100700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 10 monclient: shutdown 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 1 -- 172.21.15.123:0/4035618367 >> v1:172.21.15.151:6789/0 conn(0x7fca2c0b1730 legacy=0x7fca2c1d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 1 -- 172.21.15.123:0/4035618367 shutdown_connections 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 1 -- 172.21.15.123:0/4035618367 >> 172.21.15.123:0/4035618367 conn(0x7fca2c0617a0 msgr2=0x7fca2c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 1 -- 172.21.15.123:0/4035618367 shutdown_connections 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 1 -- 172.21.15.123:0/4035618367 wait complete. 2024-03-25T19:30:27.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:27.438+0000 7fca33725640 1 librados: shutdown 2024-03-25T19:30:27.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:34.258 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:34.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.373+0000 7f340b3c6640 10 monclient: get_monmap_and_config 2024-03-25T19:30:34.373 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.373+0000 7f340b3c6640 10 monclient: build_initial_monmap 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.373+0000 7f340b3c6640 10 monclient: monmap: 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:30:34.374861+0000 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:30:34.374861+0000 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:30:34.374 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:34.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.374+0000 7f340b3c6640 1 Processor -- start 2024-03-25T19:30:34.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.374+0000 7f340b3c6640 1 -- start start 2024-03-25T19:30:34.375 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.374+0000 7f340b3c6640 10 monclient: init 2024-03-25T19:30:34.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.375+0000 7f340b3c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:34.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.375+0000 7f340b3c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:34.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.375+0000 7f340b3c6640 10 monclient(hunting): picked mon.noname-c con 0x7f34040705b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:34.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.375+0000 7f340b3c6640 10 monclient(hunting): picked mon.noname-a con 0x7f34041402d0 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:34.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.375+0000 7f340b3c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3404149ae0 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:34.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.375+0000 7f340b3c6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34040718c0 con 0x7f34041402d0 2024-03-25T19:30:34.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.375+0000 7f340b3c6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34040714c0 con 0x7f34040705b0 2024-03-25T19:30:34.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.375+0000 7f340b3c6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3404072b50 con 0x7f3404149ae0 2024-03-25T19:30:34.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.375+0000 7f340b3c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:34.376 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.375+0000 7f340b3c6640 10 monclient(hunting): authenticate will time out at 6071.735448s 2024-03-25T19:30:34.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.381+0000 7f3409162640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f34040705b0 0x7f340413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:56790/0 (socket says 172.21.15.123:56790) 2024-03-25T19:30:34.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.381+0000 7f3409162640 1 -- 172.21.15.123:0/9032280 learned_addr learned my addr 172.21.15.123:0/9032280 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:34.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.391+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327463239 0 0) 0x7f34040714c0 con 0x7f34040705b0 2024-03-25T19:30:34.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.391+0000 7f33fbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.391+0000 7f33fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.391+0000 7f33fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.391+0000 7f33fbfff640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:30:34.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.391+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f0003520 con 0x7f34040705b0 2024-03-25T19:30:34.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.402+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504053408 0 0) 0x7f3404072b50 con 0x7f3404149ae0 2024-03-25T19:30:34.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.402+0000 7f33fbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.402+0000 7f33fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.402+0000 7f33fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.402+0000 7f33fbfff640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:30:34.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.402+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34040714c0 con 0x7f3404149ae0 2024-03-25T19:30:34.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.411+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291220117 0 0) 0x7f33f0003520 con 0x7f34040705b0 2024-03-25T19:30:34.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.411+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 >> v1:172.21.15.123:6790/0 conn(0x7f3404149ae0 legacy=0x7f340414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.411+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 >> v1:172.21.15.123:6789/0 conn(0x7f34041402d0 legacy=0x7f3404140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.411+0000 7f33fbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:34.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.411+0000 7f33fbfff640 1 monclient: found mon.noname-c 2024-03-25T19:30:34.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f33fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:34.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f33fbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:30:34.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340413dd10 con 0x7f34040705b0 2024-03-25T19:30:34.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f33fbfff640 10 monclient: _finish_auth 0 2024-03-25T19:30:34.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f33fbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f33fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f3400001690 con 0x7f34040705b0 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f340b3c6640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f340b3c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f33fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f33fbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f33fbfff640 10 monclient: dump: 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:34.413 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:34.414 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:34.414 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:34.414 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:34.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.412+0000 7f340b3c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:34.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.424+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34000017f0 con 0x7f34040705b0 2024-03-25T19:30:34.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.424+0000 7f33fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:34.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.424+0000 7f33fbfff640 1 -- 172.21.15.123:0/9032280 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f3400001c00 con 0x7f34040705b0 2024-03-25T19:30:34.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.424+0000 7f340b3c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:34.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.424+0000 7f340b3c6640 4 set_mon_vals no callback set 2024-03-25T19:30:34.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.424+0000 7f33fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.424+0000 7f33fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:34.424 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.424+0000 7f33fbfff640 10 monclient: dump: 2024-03-25T19:30:34.424 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:34.425 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:34.425 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:34.425 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:34.425 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.425 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:34.425 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:34.425 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:34.425 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:34.425 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:34.426 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.425+0000 7f340b3c6640 10 monclient: shutdown 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.425+0000 7f340b3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.425+0000 7f340b3c6640 1 -- 172.21.15.123:0/9032280 >> v1:172.21.15.151:6789/0 conn(0x7f34040705b0 legacy=0x7f340413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.425+0000 7f340b3c6640 1 -- 172.21.15.123:0/9032280 shutdown_connections 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.425+0000 7f340b3c6640 1 -- 172.21.15.123:0/9032280 >> 172.21.15.123:0/9032280 conn(0x7f34040b5d10 msgr2=0x7f34040b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.425+0000 7f340b3c6640 1 -- 172.21.15.123:0/9032280 shutdown_connections 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.425+0000 7f340b3c6640 1 -- 172.21.15.123:0/9032280 wait complete. 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.426+0000 7f340b3c6640 10 monclient: build_initial_monmap 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.426+0000 7f340b3c6640 10 monclient: monmap: 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:34.427 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.426+0000 7f340b3c6640 1 librados: starting msgr at 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.426+0000 7f340b3c6640 1 librados: starting objecter 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.426+0000 7f340b3c6640 1 Processor -- start 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.426+0000 7f340b3c6640 1 -- start start 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.426+0000 7f340b3c6640 1 librados: setting wanted keys 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.426+0000 7f340b3c6640 1 librados: calling monclient init 2024-03-25T19:30:34.428 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.426+0000 7f340b3c6640 10 monclient: init 2024-03-25T19:30:34.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.427+0000 7f340b3c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:34.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.427+0000 7f340b3c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:34.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.427+0000 7f340b3c6640 10 monclient(hunting): picked mon.noname-b con 0x7f34040705b0 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:34.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.427+0000 7f340b3c6640 10 monclient(hunting): picked mon.noname-c con 0x7f34041402d0 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:34.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.427+0000 7f340b3c6640 10 monclient(hunting): picked mon.noname-a con 0x7f3404149ae0 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:34.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.427+0000 7f340b3c6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3404141d40 con 0x7f3404149ae0 2024-03-25T19:30:34.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.427+0000 7f340b3c6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3404144450 con 0x7f34040705b0 2024-03-25T19:30:34.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.427+0000 7f340b3c6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340406d9d0 con 0x7f34041402d0 2024-03-25T19:30:34.429 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.427+0000 7f340b3c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:34.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.427+0000 7f340b3c6640 10 monclient(hunting): authenticate will time out at 6071.787750s 2024-03-25T19:30:34.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.441+0000 7f3409963640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f3404149ae0 0x7f3404143d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54192/0 (socket says 172.21.15.123:54192) 2024-03-25T19:30:34.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.441+0000 7f3409963640 1 -- 172.21.15.123:0/1712830203 learned_addr learned my addr 172.21.15.123:0/1712830203 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:34.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.441+0000 7f33f9ffb640 10 client.?.objecter ms_handle_connect 0x7f3404149ae0 2024-03-25T19:30:34.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.441+0000 7f33f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:34.451 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.451+0000 7f33f9ffb640 10 client.?.objecter ms_handle_connect 0x7f34041402d0 2024-03-25T19:30:34.451 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.451+0000 7f33f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:34.452 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.453+0000 7f33f9ffb640 10 client.?.objecter ms_handle_connect 0x7f34040705b0 2024-03-25T19:30:34.453 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.453+0000 7f33f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:34.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.458+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943257969 0 0) 0x7f340406d9d0 con 0x7f34041402d0 2024-03-25T19:30:34.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.458+0000 7f33f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.458+0000 7f33f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.458+0000 7f33f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.458+0000 7f33f9ffb640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:30:34.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.458+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e8004eb0 con 0x7f34041402d0 2024-03-25T19:30:34.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.459+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985521262 0 0) 0x7f3404141d40 con 0x7f3404149ae0 2024-03-25T19:30:34.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.459+0000 7f33f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.459+0000 7f33f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.459+0000 7f33f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.459 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.459+0000 7f33f9ffb640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:30:34.460 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.459+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f340406d9d0 con 0x7f3404149ae0 2024-03-25T19:30:34.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.463+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054902142 0 0) 0x7f3404144450 con 0x7f34040705b0 2024-03-25T19:30:34.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.463+0000 7f33f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.463+0000 7f33f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.463+0000 7f33f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.463+0000 7f33f9ffb640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:30:34.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.463+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404141d40 con 0x7f34040705b0 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.471+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1468827203 0 0) 0x7f33e8004eb0 con 0x7f34041402d0 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.471+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3404144450 con 0x7f34041402d0 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.471+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f33f4001930 con 0x7f34041402d0 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.471+0000 7f33f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.471+0000 7f33f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.471+0000 7f33f9ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:34.471 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.471+0000 7f33f9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.472+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1943797952 0 0) 0x7f340406d9d0 con 0x7f3404149ae0 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.472+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33e8004eb0 con 0x7f3404149ae0 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.472+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f33fc001b90 con 0x7f3404149ae0 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.472+0000 7f33f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.472+0000 7f33f9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.472+0000 7f33f9ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:34.472 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:34.473 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:34.473 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:34.473 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.473 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:34.473 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:34.473 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:34.473 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:34.473 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2395283535 0 0) 0x7f3404144450 con 0x7f34041402d0 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 >> v1:172.21.15.151:6789/0 conn(0x7f34040705b0 legacy=0x7f34041d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 >> v1:172.21.15.123:6789/0 conn(0x7f3404149ae0 legacy=0x7f3404143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 1 monclient: found mon.c 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340406db60 con 0x7f34041402d0 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f33f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f340b3c6640 5 monclient: authenticate success, global_id 4604 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f340b3c6640 10 monclient: _renew_subs 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f340b3c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f340b3c6640 1 -- 172.21.15.123:0/1712830203 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340413f240 con 0x7f34041402d0 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f340b3c6640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f340b3c6640 10 monclient: _renew_subs 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f340b3c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f340b3c6640 1 -- 172.21.15.123:0/1712830203 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34041d7e70 con 0x7f34041402d0 2024-03-25T19:30:34.485 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.484+0000 7f340b3c6640 1 librados: init done 2024-03-25T19:30:34.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.485+0000 7f340b3c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:34.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.485+0000 7f340b3c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:34.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.485+0000 7f340b3c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:34.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.485+0000 7f340b3c6640 1 -- 172.21.15.123:0/1712830203 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34041d8170 con 0x7f34041402d0 2024-03-25T19:30:34.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.798+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33f40035f0 con 0x7f34041402d0 2024-03-25T19:30:34.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.798+0000 7f33f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:34.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.798+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f33f4003980 con 0x7f34041402d0 2024-03-25T19:30:34.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.798+0000 7f33f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.798+0000 7f33f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:34.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.798+0000 7f33f9ffb640 10 monclient: dump: 2024-03-25T19:30:34.798 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.798+0000 7f33fb7fe640 4 set_mon_vals no callback set 2024-03-25T19:30:34.799 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.798+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f33f4003ba0 con 0x7f34041402d0 2024-03-25T19:30:34.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.799+0000 7f3409162640 10 monclient: get_auth_request con 0x7f33e806df70 auth_method 0 2024-03-25T19:30:34.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.799+0000 7f33f9ffb640 10 client.4604.objecter ms_handle_connect 0x7f33e806df70 2024-03-25T19:30:34.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.805+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f33f4053600 con 0x7f34041402d0 2024-03-25T19:30:34.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.805+0000 7f33f9ffb640 10 client.4604.objecter ms_dispatch 0x7f3404000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:34.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.805+0000 7f33f9ffb640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:34.805 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.805+0000 7f33f9ffb640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:34.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.805+0000 7f33f9ffb640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T19:30:34.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.805+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33f404f110 con 0x7f34041402d0 2024-03-25T19:30:34.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.805+0000 7f33f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:34.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:34.805+0000 7f33f9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:35.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.009+0000 7f340b3c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:35.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.009+0000 7f340b3c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:35.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.009+0000 7f340b3c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:35.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.009+0000 7f340b3c6640 1 -- 172.21.15.123:0/1712830203 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f340406dcf0 con 0x7f34041402d0 2024-03-25T19:30:35.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.042+0000 7f33f9ffb640 1 -- 172.21.15.123:0/1712830203 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f33f4051040 con 0x7f34041402d0 2024-03-25T19:30:35.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.042+0000 7f33f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:35.042 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.042+0000 7f33f9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:35.043 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:30:35.043 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:35.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 10 librados: watch_flush enter 2024-03-25T19:30:35.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 10 librados: watch_flush exit 2024-03-25T19:30:35.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T19:30:35.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 10 client.4604.objecter successfully canceled tick 2024-03-25T19:30:35.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 1 -- 172.21.15.123:0/1712830203 >> v1:172.21.15.151:6800/727684239 conn(0x7f33e806df70 legacy=0x7f33e8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:35.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 10 monclient: shutdown 2024-03-25T19:30:35.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:35.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 1 -- 172.21.15.123:0/1712830203 >> v1:172.21.15.123:6790/0 conn(0x7f34041402d0 legacy=0x7f3404143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:35.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 1 -- 172.21.15.123:0/1712830203 shutdown_connections 2024-03-25T19:30:35.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 1 -- 172.21.15.123:0/1712830203 >> 172.21.15.123:0/1712830203 conn(0x7f34040b5d10 msgr2=0x7f34040b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:35.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 1 -- 172.21.15.123:0/1712830203 shutdown_connections 2024-03-25T19:30:35.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 1 -- 172.21.15.123:0/1712830203 wait complete. 2024-03-25T19:30:35.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:35.044+0000 7f340b3c6640 1 librados: shutdown 2024-03-25T19:30:35.057 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:36.360 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:08:25.932 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:36.417 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:08:25.989 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:36.442 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:08:26.015 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:36.465 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:08:26.037 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:36.478 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:08:26.051 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:36.479 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:08:26.013 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:36.490 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:08:26.024 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:36.505 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:08:26.077 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:36.521 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:08:26.055 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:41.863 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:41.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.973+0000 7f050609a640 10 monclient: get_monmap_and_config 2024-03-25T19:30:41.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.973+0000 7f050609a640 10 monclient: build_initial_monmap 2024-03-25T19:30:41.973 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.973+0000 7f050609a640 10 monclient: monmap: 2024-03-25T19:30:41.974 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:41.974 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:41.974 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:30:41.974588+0000 2024-03-25T19:30:41.974 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:30:41.974588+0000 2024-03-25T19:30:41.974 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:41.974 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:41.974 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:41.974 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:30:41.974 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:30:41.974 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:41.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.974+0000 7f050609a640 1 Processor -- start 2024-03-25T19:30:41.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.974+0000 7f050609a640 1 -- start start 2024-03-25T19:30:41.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.974+0000 7f050609a640 10 monclient: init 2024-03-25T19:30:41.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.975+0000 7f050609a640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:41.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.975+0000 7f050609a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:41.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.975+0000 7f050609a640 10 monclient(hunting): picked mon.noname-a con 0x7f050006ea40 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:41.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.975+0000 7f050609a640 10 monclient(hunting): picked mon.noname-b con 0x7f0500071800 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:41.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.975+0000 7f050609a640 10 monclient(hunting): picked mon.noname-c con 0x7f0500071c40 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:41.975 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.975+0000 7f050609a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050006f1f0 con 0x7f050006ea40 2024-03-25T19:30:41.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.975+0000 7f050609a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0500070d00 con 0x7f0500071c40 2024-03-25T19:30:41.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.975+0000 7f050609a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05000720b0 con 0x7f0500071800 2024-03-25T19:30:41.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.975+0000 7f050609a640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:41.976 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.975+0000 7f050609a640 10 monclient(hunting): authenticate will time out at 6079.335316s 2024-03-25T19:30:41.982 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.983+0000 7f04feffd640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f0500071800 0x7f0500147070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:44572/0 (socket says 172.21.15.123:44572) 2024-03-25T19:30:41.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.983+0000 7f04feffd640 1 -- 172.21.15.123:0/2298895473 learned_addr learned my addr 172.21.15.123:0/2298895473 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:41.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.991+0000 7f04fe7fc640 1 -- 172.21.15.123:0/2298895473 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179732827 0 0) 0x7f05000720b0 con 0x7f0500071800 2024-03-25T19:30:41.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.991+0000 7f04fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:41.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.991+0000 7f04fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:41.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.991+0000 7f04fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:41.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.991+0000 7f04fe7fc640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:30:41.991 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.991+0000 7f04fe7fc640 1 -- 172.21.15.123:0/2298895473 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e8003520 con 0x7f0500071800 2024-03-25T19:30:41.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 1 -- 172.21.15.123:0/2298895473 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1699262270 0 0) 0x7f04e8003520 con 0x7f0500071800 2024-03-25T19:30:41.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 1 -- 172.21.15.123:0/2298895473 >> v1:172.21.15.151:6789/0 conn(0x7f0500071c40 legacy=0x7f050014a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 1 -- 172.21.15.123:0/2298895473 >> v1:172.21.15.123:6789/0 conn(0x7f050006ea40 legacy=0x7f050013ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 1 monclient: found mon.noname-b 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 1 -- 172.21.15.123:0/2298895473 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0500072780 con 0x7f0500071800 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 1 -- 172.21.15.123:0/2298895473 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f04f4001690 con 0x7f0500071800 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:42.000 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f04fe7fc640 10 monclient: dump: 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f050609a640 5 monclient: authenticate success, global_id 4607 2024-03-25T19:30:42.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:41.999+0000 7f050609a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:42.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.006+0000 7f04fe7fc640 1 -- 172.21.15.123:0/2298895473 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04f40017f0 con 0x7f0500071800 2024-03-25T19:30:42.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.006+0000 7f04fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:42.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.006+0000 7f04fe7fc640 1 -- 172.21.15.123:0/2298895473 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f04f4001c00 con 0x7f0500071800 2024-03-25T19:30:42.006 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.006+0000 7f050609a640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.006+0000 7f050609a640 4 set_mon_vals no callback set 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.006+0000 7f04fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.006+0000 7f04fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.006+0000 7f04fe7fc640 10 monclient: dump: 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:42.007 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:42.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.007+0000 7f050609a640 10 monclient: shutdown 2024-03-25T19:30:42.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.007+0000 7f050609a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:42.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.007+0000 7f050609a640 1 -- 172.21.15.123:0/2298895473 >> v1:172.21.15.123:6790/0 conn(0x7f0500071800 legacy=0x7f0500147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.009 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.007+0000 7f050609a640 1 -- 172.21.15.123:0/2298895473 shutdown_connections 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.007+0000 7f050609a640 1 -- 172.21.15.123:0/2298895473 >> 172.21.15.123:0/2298895473 conn(0x7f050006b630 msgr2=0x7f05000b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.007+0000 7f050609a640 1 -- 172.21.15.123:0/2298895473 shutdown_connections 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.007+0000 7f050609a640 1 -- 172.21.15.123:0/2298895473 wait complete. 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.008+0000 7f050609a640 10 monclient: build_initial_monmap 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.008+0000 7f050609a640 10 monclient: monmap: 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:30:42.010 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.008+0000 7f050609a640 1 librados: starting msgr at 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.008+0000 7f050609a640 1 librados: starting objecter 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.008+0000 7f050609a640 1 Processor -- start 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 1 -- start start 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 1 librados: setting wanted keys 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 1 librados: calling monclient init 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 10 monclient: init 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 10 monclient(hunting): picked mon.noname-b con 0x7f050006ea40 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:42.011 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 10 monclient(hunting): picked mon.noname-a con 0x7f0500071800 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:42.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 10 monclient(hunting): picked mon.noname-c con 0x7f0500071c40 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:42.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0500124190 con 0x7f0500071800 2024-03-25T19:30:42.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0500124400 con 0x7f050006ea40 2024-03-25T19:30:42.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0500141ae0 con 0x7f0500071c40 2024-03-25T19:30:42.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:42.012 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.009+0000 7f050609a640 10 monclient(hunting): authenticate will time out at 6079.369558s 2024-03-25T19:30:42.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.017+0000 7f04feffd640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f0500071800 0x7f05000a9470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:59472/0 (socket says 172.21.15.123:59472) 2024-03-25T19:30:42.017 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.017+0000 7f04feffd640 1 -- 172.21.15.123:0/1398549479 learned_addr learned my addr 172.21.15.123:0/1398549479 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:42.018 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.018+0000 7f04e7fff640 10 client.?.objecter ms_handle_connect 0x7f0500071800 2024-03-25T19:30:42.018 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.018+0000 7f04e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:42.018 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.018+0000 7f04e7fff640 10 client.?.objecter ms_handle_connect 0x7f0500071c40 2024-03-25T19:30:42.018 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.018+0000 7f04e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:42.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.024+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704607965 0 0) 0x7f0500124190 con 0x7f0500071800 2024-03-25T19:30:42.024 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.024+0000 7f04e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.024+0000 7f04e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.024+0000 7f04e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.024+0000 7f04e7fff640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:30:42.025 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.024+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec004eb0 con 0x7f0500071800 2024-03-25T19:30:42.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.030+0000 7f04e7fff640 10 client.?.objecter ms_handle_connect 0x7f050006ea40 2024-03-25T19:30:42.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.030+0000 7f04e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:42.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.031+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424780057 0 0) 0x7f0500141ae0 con 0x7f0500071c40 2024-03-25T19:30:42.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.031+0000 7f04e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.031+0000 7f04e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.031+0000 7f04e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.031+0000 7f04e7fff640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:30:42.031 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.031+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0500124190 con 0x7f0500071c40 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3247869633 0 0) 0x7f0500124190 con 0x7f0500071c40 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0500141ae0 con 0x7f0500071c40 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f04f8001b10 con 0x7f0500071c40 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 10 monclient(hunting): dump: 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:42.043 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 340190429 0 0) 0x7f04ec004eb0 con 0x7f0500071800 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0500124190 con 0x7f0500071800 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f04f4001c00 con 0x7f0500071800 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.042+0000 7f04e7fff640 10 monclient(hunting): dump: 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:42.044 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:42.045 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.045 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:42.045 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:42.045 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:42.045 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:42.045 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:42.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.045+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868083205 0 0) 0x7f0500124400 con 0x7f050006ea40 2024-03-25T19:30:42.045 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.045+0000 7f04e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.045+0000 7f04e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.045+0000 7f04e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.045+0000 7f04e7fff640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:30:42.046 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.045+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec004eb0 con 0x7f050006ea40 2024-03-25T19:30:42.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1649074917 0 0) 0x7f0500124190 con 0x7f0500071800 2024-03-25T19:30:42.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 >> v1:172.21.15.123:6790/0 conn(0x7f0500071c40 legacy=0x7f05001dff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 >> v1:172.21.15.151:6789/0 conn(0x7f050006ea40 legacy=0x7f0500122940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:42.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 1 monclient: found mon.a 2024-03-25T19:30:42.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:42.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:30:42.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0500134540 con 0x7f0500071800 2024-03-25T19:30:42.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 10 monclient: _finish_auth 0 2024-03-25T19:30:42.048 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f04e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f050609a640 5 monclient: authenticate success, global_id 4575 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f050609a640 10 monclient: _renew_subs 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f050609a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f050609a640 1 -- 172.21.15.123:0/1398549479 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05000a9b80 con 0x7f0500071800 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f050609a640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f050609a640 10 monclient: _renew_subs 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f050609a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f050609a640 1 -- 172.21.15.123:0/1398549479 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0500141660 con 0x7f0500071800 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.047+0000 7f050609a640 1 librados: init done 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.048+0000 7f050609a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.048+0000 7f050609a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:42.049 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.048+0000 7f050609a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:30:42.050 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.048+0000 7f050609a640 1 -- 172.21.15.123:0/1398549479 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0500141d20 con 0x7f0500071800 2024-03-25T19:30:42.339 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.339+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04f4004a90 con 0x7f0500071800 2024-03-25T19:30:42.339 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.339+0000 7f04e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:42.339 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.339+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f04f4004e20 con 0x7f0500071800 2024-03-25T19:30:42.339 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.339+0000 7f04e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.339 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.339+0000 7f04e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:42.339 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.339+0000 7f04e7fff640 10 monclient: dump: 2024-03-25T19:30:42.339 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:42.339 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:42.340 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:42.340 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:42.340 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.340 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:42.340 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:42.340 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:42.340 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:42.340 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:42.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.339+0000 7f04fdffb640 4 set_mon_vals no callback set 2024-03-25T19:30:42.341 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.340+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f04f4005040 con 0x7f0500071800 2024-03-25T19:30:42.341 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.340+0000 7f04ff7fe640 10 monclient: get_auth_request con 0x7f04ec06dfc0 auth_method 0 2024-03-25T19:30:42.341 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.340+0000 7f04e7fff640 10 client.4575.objecter ms_handle_connect 0x7f04ec06dfc0 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.345+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f04f4054960 con 0x7f0500071800 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.345+0000 7f04e7fff640 10 client.4575.objecter ms_dispatch 0x7f0500000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.345+0000 7f04e7fff640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.345+0000 7f04e7fff640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.345+0000 7f04e7fff640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.345+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04f4050470 con 0x7f0500071800 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.345+0000 7f04e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:42.346 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.345+0000 7f04e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:42.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.595+0000 7f050609a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.595+0000 7f050609a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.595+0000 7f050609a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:30:42.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.595+0000 7f050609a640 1 -- 172.21.15.123:0/1398549479 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05000627b0 con 0x7f0500071800 2024-03-25T19:30:42.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.625+0000 7f04e7fff640 1 -- 172.21.15.123:0/1398549479 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f04f40523a0 con 0x7f0500071800 2024-03-25T19:30:42.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.625+0000 7f04e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.625+0000 7f04e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:42.625 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:30:42.625 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:42.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 10 librados: watch_flush enter 2024-03-25T19:30:42.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 10 librados: watch_flush exit 2024-03-25T19:30:42.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T19:30:42.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 10 client.4575.objecter successfully canceled tick 2024-03-25T19:30:42.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 1 -- 172.21.15.123:0/1398549479 >> v1:172.21.15.151:6800/727684239 conn(0x7f04ec06dfc0 legacy=0x7f04ec070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 10 monclient: shutdown 2024-03-25T19:30:42.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:42.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 1 -- 172.21.15.123:0/1398549479 >> v1:172.21.15.123:6789/0 conn(0x7f0500071800 legacy=0x7f05000a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 1 -- 172.21.15.123:0/1398549479 shutdown_connections 2024-03-25T19:30:42.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 1 -- 172.21.15.123:0/1398549479 >> 172.21.15.123:0/1398549479 conn(0x7f050006b630 msgr2=0x7f05000af620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:42.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 1 -- 172.21.15.123:0/1398549479 shutdown_connections 2024-03-25T19:30:42.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 1 -- 172.21.15.123:0/1398549479 wait complete. 2024-03-25T19:30:42.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:42.627+0000 7f050609a640 1 librados: shutdown 2024-03-25T19:30:42.638 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:49.445 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:49.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.558+0000 7f6922b3a640 10 monclient: get_monmap_and_config 2024-03-25T19:30:49.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.558+0000 7f6922b3a640 10 monclient: build_initial_monmap 2024-03-25T19:30:49.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.558+0000 7f6922b3a640 10 monclient: monmap: 2024-03-25T19:30:49.559 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:49.559 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:49.559 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:30:49.560685+0000 2024-03-25T19:30:49.560 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:30:49.560685+0000 2024-03-25T19:30:49.560 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:49.560 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:49.560 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:49.560 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:30:49.560 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:30:49.560 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:49.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.559+0000 7f6922b3a640 1 Processor -- start 2024-03-25T19:30:49.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.559+0000 7f6922b3a640 1 -- start start 2024-03-25T19:30:49.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.559+0000 7f6922b3a640 10 monclient: init 2024-03-25T19:30:49.561 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.560+0000 7f6922b3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:49.561 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.560+0000 7f6922b3a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:49.561 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.560+0000 7f6922b3a640 10 monclient(hunting): picked mon.noname-b con 0x7f691c0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:49.561 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.560+0000 7f6922b3a640 10 monclient(hunting): picked mon.noname-c con 0x7f691c0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:49.561 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.560+0000 7f6922b3a640 10 monclient(hunting): picked mon.noname-a con 0x7f691c14be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:49.561 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.560+0000 7f6922b3a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0af940 con 0x7f691c14be90 2024-03-25T19:30:49.561 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.560+0000 7f6922b3a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0b0a70 con 0x7f691c0b1730 2024-03-25T19:30:49.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.560+0000 7f6922b3a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0b0d50 con 0x7f691c0b4d60 2024-03-25T19:30:49.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.560+0000 7f6922b3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:49.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.560+0000 7f6922b3a640 10 monclient(hunting): authenticate will time out at 6086.921621s 2024-03-25T19:30:49.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.570+0000 7f69210d7640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f691c14be90 0x7f691c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:34760/0 (socket says 172.21.15.123:34760) 2024-03-25T19:30:49.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.570+0000 7f69210d7640 1 -- 172.21.15.123:0/801646806 learned_addr learned my addr 172.21.15.123:0/801646806 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:49.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.581+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626771447 0 0) 0x7f691c0af940 con 0x7f691c14be90 2024-03-25T19:30:49.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.581+0000 7f691b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.581+0000 7f691b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.581+0000 7f691b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.581+0000 7f691b7fe640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:30:49.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.581+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690c003520 con 0x7f691c14be90 2024-03-25T19:30:49.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.586+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465480001 0 0) 0x7f691c0b0d50 con 0x7f691c0b4d60 2024-03-25T19:30:49.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.586+0000 7f691b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.586+0000 7f691b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.586+0000 7f691b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.586+0000 7f691b7fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:30:49.587 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.586+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c0af940 con 0x7f691c0b4d60 2024-03-25T19:30:49.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.592+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524669967 0 0) 0x7f691c0b0a70 con 0x7f691c0b1730 2024-03-25T19:30:49.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.592+0000 7f691b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.592+0000 7f691b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.592+0000 7f691b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.592+0000 7f691b7fe640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:30:49.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.592+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c0b0d50 con 0x7f691c0b1730 2024-03-25T19:30:49.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3018230521 0 0) 0x7f690c003520 con 0x7f691c14be90 2024-03-25T19:30:49.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 >> v1:172.21.15.123:6790/0 conn(0x7f691c0b4d60 legacy=0x7f691c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 >> v1:172.21.15.151:6789/0 conn(0x7f691c0b1730 legacy=0x7f691c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:49.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:30:49.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:49.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:30:49.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c0b1030 con 0x7f691c14be90 2024-03-25T19:30:49.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:49.598 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6914001a10 con 0x7f691c14be90 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f691b7fe640 10 monclient: dump: 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:49.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f6922b3a640 5 monclient: authenticate success, global_id 4578 2024-03-25T19:30:49.600 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.596+0000 7f6922b3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.624+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6914001b70 con 0x7f691c14be90 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.624+0000 7f691b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.624+0000 7f691b7fe640 1 -- 172.21.15.123:0/801646806 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6914001f80 con 0x7f691c14be90 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.624+0000 7f6922b3a640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.624+0000 7f6922b3a640 4 set_mon_vals no callback set 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.624+0000 7f691b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.624+0000 7f691b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.624+0000 7f691b7fe640 10 monclient: dump: 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:49.625 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:49.626 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.626 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:49.626 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:49.626 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:49.626 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:49.626 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:49.626 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.625+0000 7f6922b3a640 10 monclient: shutdown 2024-03-25T19:30:49.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.625+0000 7f6922b3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:49.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.625+0000 7f6922b3a640 1 -- 172.21.15.123:0/801646806 >> v1:172.21.15.123:6789/0 conn(0x7f691c14be90 legacy=0x7f691c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.625+0000 7f6922b3a640 1 -- 172.21.15.123:0/801646806 shutdown_connections 2024-03-25T19:30:49.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.625+0000 7f6922b3a640 1 -- 172.21.15.123:0/801646806 >> 172.21.15.123:0/801646806 conn(0x7f691c0617a0 msgr2=0x7f691c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:49.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.625+0000 7f6922b3a640 1 -- 172.21.15.123:0/801646806 shutdown_connections 2024-03-25T19:30:49.627 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.625+0000 7f6922b3a640 1 -- 172.21.15.123:0/801646806 wait complete. 2024-03-25T19:30:49.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.626+0000 7f6922b3a640 10 monclient: build_initial_monmap 2024-03-25T19:30:49.628 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.626+0000 7f6922b3a640 10 monclient: monmap: 2024-03-25T19:30:49.628 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:49.628 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.626+0000 7f6922b3a640 1 librados: starting msgr at 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.626+0000 7f6922b3a640 1 librados: starting objecter 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.626+0000 7f6922b3a640 1 Processor -- start 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.626+0000 7f6922b3a640 1 -- start start 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.626+0000 7f6922b3a640 1 librados: setting wanted keys 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.626+0000 7f6922b3a640 1 librados: calling monclient init 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.626+0000 7f6922b3a640 10 monclient: init 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.627+0000 7f6922b3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:49.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.627+0000 7f6922b3a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:49.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.627+0000 7f6922b3a640 10 monclient(hunting): picked mon.noname-b con 0x7f691c0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:49.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.627+0000 7f6922b3a640 10 monclient(hunting): picked mon.noname-a con 0x7f691c0b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:49.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.627+0000 7f6922b3a640 10 monclient(hunting): picked mon.noname-c con 0x7f691c14be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:49.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.627+0000 7f6922b3a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c142280 con 0x7f691c0b4d60 2024-03-25T19:30:49.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.627+0000 7f6922b3a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c06e040 con 0x7f691c0b1730 2024-03-25T19:30:49.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.627+0000 7f6922b3a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c136fb0 con 0x7f691c14be90 2024-03-25T19:30:49.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.627+0000 7f6922b3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:49.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.627+0000 7f6922b3a640 10 monclient(hunting): authenticate will time out at 6086.988659s 2024-03-25T19:30:49.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.648+0000 7f69210d7640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f691c14be90 0x7f691c1368a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:58194/0 (socket says 172.21.15.123:58194) 2024-03-25T19:30:49.648 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.648+0000 7f69210d7640 1 -- 172.21.15.123:0/35016308 learned_addr learned my addr 172.21.15.123:0/35016308 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:49.649 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.648+0000 7f69197fa640 10 client.?.objecter ms_handle_connect 0x7f691c14be90 2024-03-25T19:30:49.649 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.648+0000 7f69197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:49.653 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.653+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763468822 0 0) 0x7f691c136fb0 con 0x7f691c14be90 2024-03-25T19:30:49.653 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.653+0000 7f69197fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.653+0000 7f69197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.653+0000 7f69197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.653+0000 7f69197fa640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:30:49.654 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.653+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6900004eb0 con 0x7f691c14be90 2024-03-25T19:30:49.655 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.654+0000 7f69197fa640 10 client.?.objecter ms_handle_connect 0x7f691c0b1730 2024-03-25T19:30:49.655 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.654+0000 7f69197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:49.657 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.656+0000 7f69197fa640 10 client.?.objecter ms_handle_connect 0x7f691c0b4d60 2024-03-25T19:30:49.657 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.656+0000 7f69197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:49.661 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.660+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849357472 0 0) 0x7f691c06e040 con 0x7f691c0b1730 2024-03-25T19:30:49.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.660+0000 7f69197fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.660+0000 7f69197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.660+0000 7f69197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.660+0000 7f69197fa640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:30:49.662 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.660+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c136fb0 con 0x7f691c0b1730 2024-03-25T19:30:49.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.666+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855229972 0 0) 0x7f691c142280 con 0x7f691c0b4d60 2024-03-25T19:30:49.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.667+0000 7f69197fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.667 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.667+0000 7f69197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.667+0000 7f69197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.667+0000 7f69197fa640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:30:49.668 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.667+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c06e040 con 0x7f691c0b4d60 2024-03-25T19:30:49.689 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.688+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415991286 0 0) 0x7f6900004eb0 con 0x7f691c14be90 2024-03-25T19:30:49.689 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.688+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f691c142280 con 0x7f691c14be90 2024-03-25T19:30:49.689 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.688+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6914001ec0 con 0x7f691c14be90 2024-03-25T19:30:49.689 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.688+0000 7f69197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.689 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.688+0000 7f69197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:49.689 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.688+0000 7f69197fa640 10 monclient(hunting): dump: 2024-03-25T19:30:49.689 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:49.689 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:49.689 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:49.689 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:49.690 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.690 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:49.690 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:49.690 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:49.690 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:49.690 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:49.690 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.688+0000 7f69197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:49.690 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.689+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 222611431 0 0) 0x7f691c06e040 con 0x7f691c0b4d60 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.689+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6900004eb0 con 0x7f691c0b4d60 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.689+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6908001830 con 0x7f691c0b4d60 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.689+0000 7f69197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.689+0000 7f69197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.689+0000 7f69197fa640 10 monclient(hunting): dump: 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:49.691 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:49.692 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.691+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 741002775 0 0) 0x7f691c136fb0 con 0x7f691c0b1730 2024-03-25T19:30:49.692 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.692+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f691c06e040 con 0x7f691c0b1730 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.692+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6910001970 con 0x7f691c0b1730 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.692+0000 7f69197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.692+0000 7f69197fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.692+0000 7f69197fa640 10 monclient(hunting): dump: 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:49.693 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:49.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2943086248 0 0) 0x7f691c142280 con 0x7f691c14be90 2024-03-25T19:30:49.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 >> v1:172.21.15.151:6789/0 conn(0x7f691c0b1730 legacy=0x7f691c13a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 >> v1:172.21.15.123:6789/0 conn(0x7f691c0b4d60 legacy=0x7f691c136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 10 monclient: _finish_hunting 0 2024-03-25T19:30:49.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 1 monclient: found mon.c 2024-03-25T19:30:49.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:49.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:49.704 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c141aa0 con 0x7f691c14be90 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 10 monclient: _finish_auth 0 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 10 monclient: _check_auth_tickets 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f69197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f6922b3a640 5 monclient: authenticate success, global_id 4616 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f6922b3a640 10 monclient: _renew_subs 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f6922b3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f6922b3a640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c142710 con 0x7f691c14be90 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f6922b3a640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f6922b3a640 10 monclient: _renew_subs 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f6922b3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f6922b3a640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c1408e0 con 0x7f691c14be90 2024-03-25T19:30:49.705 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.703+0000 7f6922b3a640 1 librados: init done 2024-03-25T19:30:49.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.704+0000 7f6922b3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:49.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.704+0000 7f6922b3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:49.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.704+0000 7f6922b3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:49.706 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:49.704+0000 7f6922b3a640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c0627b0 con 0x7f691c14be90 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.028+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6914004e10 con 0x7f691c14be90 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.028+0000 7f69197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.028+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6914002100 con 0x7f691c14be90 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.028+0000 7f69197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.028+0000 7f69197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.028+0000 7f69197fa640 10 monclient: dump: 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:50.029 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:50.030 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:50.030 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:50.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.028+0000 7f691affd640 4 set_mon_vals no callback set 2024-03-25T19:30:50.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.029+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f691401af90 con 0x7f691c14be90 2024-03-25T19:30:50.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.029+0000 7f69208d6640 10 monclient: get_auth_request con 0x7f690006e0b0 auth_method 0 2024-03-25T19:30:50.030 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.030+0000 7f69197fa640 10 client.4616.objecter ms_handle_connect 0x7f690006e0b0 2024-03-25T19:30:50.034 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.033+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f6914054a80 con 0x7f691c14be90 2024-03-25T19:30:50.034 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.033+0000 7f69197fa640 10 client.4616.objecter ms_dispatch 0x7f691c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.033+0000 7f69197fa640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.033+0000 7f69197fa640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.033+0000 7f69197fa640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-25T19:30:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.033+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69140503c0 con 0x7f691c14be90 2024-03-25T19:30:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.033+0000 7f69197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:50.035 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.034+0000 7f69197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:50.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.263+0000 7f6922b3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:50.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.263+0000 7f6922b3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:50.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.263+0000 7f6922b3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:50.265 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.263+0000 7f6922b3a640 1 -- 172.21.15.123:0/35016308 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f691c142430 con 0x7f691c14be90 2024-03-25T19:30:50.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.305+0000 7f69197fa640 1 -- 172.21.15.123:0/35016308 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f691c142430 con 0x7f691c14be90 2024-03-25T19:30:50.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.305+0000 7f69197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:50.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.305+0000 7f69197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:50.306 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:30:50.306 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:50.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 10 librados: watch_flush enter 2024-03-25T19:30:50.308 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 10 librados: watch_flush exit 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 10 client.4616.objecter successfully canceled tick 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 1 -- 172.21.15.123:0/35016308 >> v1:172.21.15.151:6800/727684239 conn(0x7f690006e0b0 legacy=0x7f6900070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 10 monclient: shutdown 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 1 -- 172.21.15.123:0/35016308 >> v1:172.21.15.123:6790/0 conn(0x7f691c14be90 legacy=0x7f691c1368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 1 -- 172.21.15.123:0/35016308 shutdown_connections 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 1 -- 172.21.15.123:0/35016308 >> 172.21.15.123:0/35016308 conn(0x7f691c0617a0 msgr2=0x7f691c13c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 1 -- 172.21.15.123:0/35016308 shutdown_connections 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.307+0000 7f6922b3a640 1 -- 172.21.15.123:0/35016308 wait complete. 2024-03-25T19:30:50.309 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:50.308+0000 7f6922b3a640 1 librados: shutdown 2024-03-25T19:30:50.320 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:56.332 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:08:45.905 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:56.337 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:08:45.909 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:56.342 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:08:45.914 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:56.349 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:08:45.922 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:56.377 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:08:45.949 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:56.393 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:08:45.926 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:56.395 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:08:45.968 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:56.396 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:08:45.930 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:56.423 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:08:45.957 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:57.125 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.241+0000 7fc67ef65640 10 monclient: get_monmap_and_config 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.241+0000 7fc67ef65640 10 monclient: build_initial_monmap 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.241+0000 7fc67ef65640 10 monclient: monmap: 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:30:57.242344+0000 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:30:57.242344+0000 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:30:57.241 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:30:57.242 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:57.242 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.242+0000 7fc67ef65640 1 Processor -- start 2024-03-25T19:30:57.242 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.242+0000 7fc67ef65640 1 -- start start 2024-03-25T19:30:57.242 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.242+0000 7fc67ef65640 10 monclient: init 2024-03-25T19:30:57.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.243+0000 7fc67ef65640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:57.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.243+0000 7fc67ef65640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:57.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.243+0000 7fc67ef65640 10 monclient(hunting): picked mon.noname-a con 0x7fc678061830 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:57.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.243+0000 7fc67ef65640 10 monclient(hunting): picked mon.noname-b con 0x7fc6780ae840 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:57.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.243+0000 7fc67ef65640 10 monclient(hunting): picked mon.noname-c con 0x7fc67813fbd0 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:57.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.243+0000 7fc67ef65640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6780ae1c0 con 0x7fc678061830 2024-03-25T19:30:57.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.243+0000 7fc67ef65640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67806f7d0 con 0x7fc67813fbd0 2024-03-25T19:30:57.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.243+0000 7fc67ef65640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67806fa40 con 0x7fc6780ae840 2024-03-25T19:30:57.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.243+0000 7fc67ef65640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:57.244 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.243+0000 7fc67ef65640 10 monclient(hunting): authenticate will time out at 6094.603330s 2024-03-25T19:30:57.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.246+0000 7fc67d502640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fc67813fbd0 0x7fc678145f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:59692/0 (socket says 172.21.15.123:59692) 2024-03-25T19:30:57.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.246+0000 7fc67d502640 1 -- 172.21.15.123:0/3195281436 learned_addr learned my addr 172.21.15.123:0/3195281436 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:57.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.251+0000 7fc6777fe640 1 -- 172.21.15.123:0/3195281436 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745904089 0 0) 0x7fc67806f7d0 con 0x7fc67813fbd0 2024-03-25T19:30:57.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.251+0000 7fc6777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.251+0000 7fc6777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.251+0000 7fc6777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.251+0000 7fc6777fe640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:30:57.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.251+0000 7fc6777fe640 1 -- 172.21.15.123:0/3195281436 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc668003520 con 0x7fc67813fbd0 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.259+0000 7fc6777fe640 1 -- 172.21.15.123:0/3195281436 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4033441390 0 0) 0x7fc668003520 con 0x7fc67813fbd0 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.259+0000 7fc6777fe640 1 -- 172.21.15.123:0/3195281436 >> v1:172.21.15.123:6790/0 conn(0x7fc6780ae840 legacy=0x7fc678145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.259+0000 7fc6777fe640 1 -- 172.21.15.123:0/3195281436 >> v1:172.21.15.123:6789/0 conn(0x7fc678061830 legacy=0x7fc67813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.259+0000 7fc6777fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.259+0000 7fc6777fe640 1 monclient: found mon.noname-c 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.259+0000 7fc6777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc6777fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc6777fe640 1 -- 172.21.15.123:0/3195281436 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc678073040 con 0x7fc67813fbd0 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc6777fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc6777fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:57.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc6777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc6777fe640 1 -- 172.21.15.123:0/3195281436 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc670001a10 con 0x7fc67813fbd0 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc67ef65640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc67ef65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc6777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc6777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc6777fe640 10 monclient: dump: 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:57.261 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:57.262 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:57.262 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.260+0000 7fc67ef65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:57.263 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc6777fe640 1 -- 172.21.15.123:0/3195281436 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc670001b70 con 0x7fc67813fbd0 2024-03-25T19:30:57.263 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc6777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc6777fe640 1 -- 172.21.15.123:0/3195281436 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc670001fc0 con 0x7fc67813fbd0 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc67ef65640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc67ef65640 4 set_mon_vals no callback set 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc6777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc6777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc6777fe640 10 monclient: dump: 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:57.264 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc67ef65640 10 monclient: shutdown 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc67ef65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.264+0000 7fc67ef65640 1 -- 172.21.15.123:0/3195281436 >> v1:172.21.15.151:6789/0 conn(0x7fc67813fbd0 legacy=0x7fc678145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.265+0000 7fc67ef65640 1 -- 172.21.15.123:0/3195281436 shutdown_connections 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.265+0000 7fc67ef65640 1 -- 172.21.15.123:0/3195281436 >> 172.21.15.123:0/3195281436 conn(0x7fc6780b0ef0 msgr2=0x7fc6780b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.265+0000 7fc67ef65640 1 -- 172.21.15.123:0/3195281436 shutdown_connections 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.265+0000 7fc67ef65640 1 -- 172.21.15.123:0/3195281436 wait complete. 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.265+0000 7fc67ef65640 10 monclient: build_initial_monmap 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.265+0000 7fc67ef65640 10 monclient: monmap: 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:30:57.266 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.266+0000 7fc67ef65640 1 librados: starting msgr at 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.266+0000 7fc67ef65640 1 librados: starting objecter 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.266+0000 7fc67ef65640 1 Processor -- start 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.266+0000 7fc67ef65640 1 -- start start 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.266+0000 7fc67ef65640 1 librados: setting wanted keys 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.266+0000 7fc67ef65640 1 librados: calling monclient init 2024-03-25T19:30:57.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.266+0000 7fc67ef65640 10 monclient: init 2024-03-25T19:30:57.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.267+0000 7fc67ef65640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:57.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.267+0000 7fc67ef65640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:57.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.267+0000 7fc67ef65640 10 monclient(hunting): picked mon.noname-c con 0x7fc678061830 addr v1:172.21.15.123:6790/0 2024-03-25T19:30:57.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.267+0000 7fc67ef65640 10 monclient(hunting): picked mon.noname-a con 0x7fc6780ae840 addr v1:172.21.15.123:6789/0 2024-03-25T19:30:57.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.267+0000 7fc67ef65640 10 monclient(hunting): picked mon.noname-b con 0x7fc67813fbd0 addr v1:172.21.15.151:6789/0 2024-03-25T19:30:57.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.267+0000 7fc67ef65640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6781d8260 con 0x7fc6780ae840 2024-03-25T19:30:57.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.267+0000 7fc67ef65640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67813ac40 con 0x7fc67813fbd0 2024-03-25T19:30:57.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.267+0000 7fc67ef65640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6781d7350 con 0x7fc678061830 2024-03-25T19:30:57.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.267+0000 7fc67ef65640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:57.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.267+0000 7fc67ef65640 10 monclient(hunting): authenticate will time out at 6094.627728s 2024-03-25T19:30:57.277 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.278+0000 7fc677fff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fc6780ae840 0x7fc6781d7b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:41112/0 (socket says 172.21.15.123:41112) 2024-03-25T19:30:57.277 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.278+0000 7fc677fff640 1 -- 172.21.15.123:0/3572019480 learned_addr learned my addr 172.21.15.123:0/3572019480 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:30:57.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.279+0000 7fc6757fa640 10 client.?.objecter ms_handle_connect 0x7fc6780ae840 2024-03-25T19:30:57.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.279+0000 7fc6757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:57.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.280+0000 7fc6757fa640 10 client.?.objecter ms_handle_connect 0x7fc678061830 2024-03-25T19:30:57.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.280+0000 7fc6757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:57.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.281+0000 7fc6757fa640 10 client.?.objecter ms_handle_connect 0x7fc67813fbd0 2024-03-25T19:30:57.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.281+0000 7fc6757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:57.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.286+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146108156 0 0) 0x7fc6781d7350 con 0x7fc678061830 2024-03-25T19:30:57.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.286+0000 7fc6757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.286+0000 7fc6757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.286+0000 7fc6757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.286+0000 7fc6757fa640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:30:57.286 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.286+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc65c004eb0 con 0x7fc678061830 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.293+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2696845437 0 0) 0x7fc65c004eb0 con 0x7fc678061830 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.293+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6781d7350 con 0x7fc678061830 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.293+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc66c001890 con 0x7fc678061830 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.293+0000 7fc6757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.293+0000 7fc6757fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.293+0000 7fc6757fa640 10 monclient(hunting): dump: 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:57.294 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:57.295 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:57.295 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:57.295 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:57.295 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.293+0000 7fc6757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:57.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.299+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030631124 0 0) 0x7fc67813ac40 con 0x7fc67813fbd0 2024-03-25T19:30:57.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.299+0000 7fc6757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.299 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.299+0000 7fc6757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.299+0000 7fc6757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.299+0000 7fc6757fa640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:30:57.300 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.299+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc65c004eb0 con 0x7fc67813fbd0 2024-03-25T19:30:57.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.303+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347370344 0 0) 0x7fc6781d8260 con 0x7fc6780ae840 2024-03-25T19:30:57.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.303+0000 7fc6757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.303+0000 7fc6757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.303+0000 7fc6757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.303+0000 7fc6757fa640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:30:57.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.303+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc67813ac40 con 0x7fc6780ae840 2024-03-25T19:30:57.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3885924053 0 0) 0x7fc6781d7350 con 0x7fc678061830 2024-03-25T19:30:57.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 >> v1:172.21.15.151:6789/0 conn(0x7fc67813fbd0 legacy=0x7fc6781d46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 >> v1:172.21.15.123:6789/0 conn(0x7fc6780ae840 legacy=0x7fc6781d7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 10 monclient: _finish_hunting 0 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 1 monclient: found mon.c 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6781d7630 con 0x7fc678061830 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 10 monclient: _finish_auth 0 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 10 monclient: _check_auth_tickets 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc6757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc67ef65640 5 monclient: authenticate success, global_id 4619 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc67ef65640 10 monclient: _renew_subs 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc67ef65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc67ef65640 1 -- 172.21.15.123:0/3572019480 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6781d4df0 con 0x7fc678061830 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc67ef65640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc67ef65640 10 monclient: _renew_subs 2024-03-25T19:30:57.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc67ef65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:57.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc67ef65640 1 -- 172.21.15.123:0/3572019480 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc678139800 con 0x7fc678061830 2024-03-25T19:30:57.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.304+0000 7fc67ef65640 1 librados: init done 2024-03-25T19:30:57.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.305+0000 7fc67ef65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:57.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.305+0000 7fc67ef65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:57.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.305+0000 7fc67ef65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:57.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.305+0000 7fc67ef65640 1 -- 172.21.15.123:0/3572019480 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6780627b0 con 0x7fc678061830 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.603+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc66c0019f0 con 0x7fc678061830 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.603+0000 7fc6757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.603+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fc66c0037b0 con 0x7fc678061830 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.603+0000 7fc6757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.604+0000 7fc6757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.604+0000 7fc6757fa640 10 monclient: dump: 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:30:57.604 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.605 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:30:57.605 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:30:57.605 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:30:57.605 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:30:57.605 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:30:57.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.604+0000 7fc676ffd640 4 set_mon_vals no callback set 2024-03-25T19:30:57.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.604+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fc66c0039d0 con 0x7fc678061830 2024-03-25T19:30:57.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.605+0000 7fc677fff640 10 monclient: get_auth_request con 0x7fc65c06de50 auth_method 0 2024-03-25T19:30:57.606 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.605+0000 7fc6757fa640 10 client.4619.objecter ms_handle_connect 0x7fc65c06de50 2024-03-25T19:30:57.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.611+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fc66c053610 con 0x7fc678061830 2024-03-25T19:30:57.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.611+0000 7fc6757fa640 10 client.4619.objecter ms_dispatch 0x7fc678000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:57.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.611+0000 7fc6757fa640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:57.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.611+0000 7fc6757fa640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:57.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.612+0000 7fc6757fa640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-25T19:30:57.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.612+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc66c04ef50 con 0x7fc678061830 2024-03-25T19:30:57.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.612+0000 7fc6757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:57.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.612+0000 7fc6757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.827+0000 7fc67ef65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.827+0000 7fc67ef65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.827+0000 7fc67ef65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.827+0000 7fc67ef65640 1 -- 172.21.15.123:0/3572019480 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc678069f40 con 0x7fc678061830 2024-03-25T19:30:57.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.921+0000 7fc6757fa640 1 -- 172.21.15.123:0/3572019480 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fc66c051050 con 0x7fc678061830 2024-03-25T19:30:57.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.921+0000 7fc6757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:57.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.921+0000 7fc6757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:57.922 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:30:57.922 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:57.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 10 librados: watch_flush enter 2024-03-25T19:30:57.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 10 librados: watch_flush exit 2024-03-25T19:30:57.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-25T19:30:57.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 10 client.4619.objecter successfully canceled tick 2024-03-25T19:30:57.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 1 -- 172.21.15.123:0/3572019480 >> v1:172.21.15.151:6800/727684239 conn(0x7fc65c06de50 legacy=0x7fc65c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 10 monclient: shutdown 2024-03-25T19:30:57.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:57.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 1 -- 172.21.15.123:0/3572019480 >> v1:172.21.15.123:6790/0 conn(0x7fc678061830 legacy=0x7fc6781389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 1 -- 172.21.15.123:0/3572019480 shutdown_connections 2024-03-25T19:30:57.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 1 -- 172.21.15.123:0/3572019480 >> 172.21.15.123:0/3572019480 conn(0x7fc6780b0ef0 msgr2=0x7fc67813f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:57.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 1 -- 172.21.15.123:0/3572019480 shutdown_connections 2024-03-25T19:30:57.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 1 -- 172.21.15.123:0/3572019480 wait complete. 2024-03-25T19:30:57.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:30:57.924+0000 7fc67ef65640 1 librados: shutdown 2024-03-25T19:30:57.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:04.742 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:04.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.841+0000 7fd25a515640 10 monclient: get_monmap_and_config 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.841+0000 7fd25a515640 10 monclient: build_initial_monmap 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.841+0000 7fd25a515640 10 monclient: monmap: 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:31:04.841940+0000 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:31:04.841940+0000 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:31:04.841 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:31:04.842 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:04.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.841+0000 7fd25a515640 1 Processor -- start 2024-03-25T19:31:04.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.841+0000 7fd25a515640 1 -- start start 2024-03-25T19:31:04.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.841+0000 7fd25a515640 10 monclient: init 2024-03-25T19:31:04.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.842+0000 7fd25a515640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:04.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.842+0000 7fd25a515640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:04.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.842+0000 7fd25a515640 10 monclient(hunting): picked mon.noname-b con 0x7fd2540a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:04.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.842+0000 7fd25a515640 10 monclient(hunting): picked mon.noname-c con 0x7fd25413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:04.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.842+0000 7fd25a515640 10 monclient(hunting): picked mon.noname-a con 0x7fd2541430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:04.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.842+0000 7fd25a515640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd254142520 con 0x7fd2541430c0 2024-03-25T19:31:04.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.842+0000 7fd25a515640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2540b28a0 con 0x7fd25413f710 2024-03-25T19:31:04.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.842+0000 7fd25a515640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2540b2a30 con 0x7fd2540a98f0 2024-03-25T19:31:04.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.842+0000 7fd25a515640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:04.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.843+0000 7fd25a515640 10 monclient(hunting): authenticate will time out at 6102.202879s 2024-03-25T19:31:04.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.853+0000 7fd258ab2640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fd2541430c0 0x7fd25414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:45870/0 (socket says 172.21.15.123:45870) 2024-03-25T19:31:04.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.854+0000 7fd258ab2640 1 -- 172.21.15.123:0/1141674320 learned_addr learned my addr 172.21.15.123:0/1141674320 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:04.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.864+0000 7fd252ffd640 1 -- 172.21.15.123:0/1141674320 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470245770 0 0) 0x7fd254142520 con 0x7fd2541430c0 2024-03-25T19:31:04.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.864+0000 7fd252ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.864+0000 7fd252ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.864+0000 7fd252ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.865 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.864+0000 7fd252ffd640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:31:04.865 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.864+0000 7fd252ffd640 1 -- 172.21.15.123:0/1141674320 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd23c003520 con 0x7fd2541430c0 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 1 -- 172.21.15.123:0/1141674320 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3112236178 0 0) 0x7fd23c003520 con 0x7fd2541430c0 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 1 -- 172.21.15.123:0/1141674320 >> v1:172.21.15.123:6790/0 conn(0x7fd2540a98f0 legacy=0x7fd2540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 1 -- 172.21.15.123:0/1141674320 >> v1:172.21.15.151:6789/0 conn(0x7fd25413f710 legacy=0x7fd2541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 1 monclient: found mon.noname-a 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 1 -- 172.21.15.123:0/1141674320 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2540b44c0 con 0x7fd2541430c0 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:04.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 1 -- 172.21.15.123:0/1141674320 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd24c001a10 con 0x7fd2541430c0 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd25a515640 5 monclient: authenticate success, global_id 4590 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd25a515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.890+0000 7fd252ffd640 10 monclient: dump: 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:04.892 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:04.893 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:04.893 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:04.893 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:04.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.891+0000 7fd25a515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:04.909 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.909+0000 7fd252ffd640 1 -- 172.21.15.123:0/1141674320 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd24c001be0 con 0x7fd2541430c0 2024-03-25T19:31:04.909 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.909+0000 7fd252ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:04.909 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.909+0000 7fd252ffd640 1 -- 172.21.15.123:0/1141674320 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd24c002030 con 0x7fd2541430c0 2024-03-25T19:31:04.909 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.909+0000 7fd25a515640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:04.909 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.909+0000 7fd25a515640 4 set_mon_vals no callback set 2024-03-25T19:31:04.909 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.909+0000 7fd252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.909 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.909+0000 7fd252ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.909+0000 7fd252ffd640 10 monclient: dump: 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:04.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.910+0000 7fd25a515640 10 monclient: shutdown 2024-03-25T19:31:04.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.910+0000 7fd25a515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:04.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.910+0000 7fd25a515640 1 -- 172.21.15.123:0/1141674320 >> v1:172.21.15.123:6789/0 conn(0x7fd2541430c0 legacy=0x7fd25414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.910+0000 7fd25a515640 1 -- 172.21.15.123:0/1141674320 shutdown_connections 2024-03-25T19:31:04.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.910+0000 7fd25a515640 1 -- 172.21.15.123:0/1141674320 >> 172.21.15.123:0/1141674320 conn(0x7fd2540b7ea0 msgr2=0x7fd2540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:04.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.910+0000 7fd25a515640 1 -- 172.21.15.123:0/1141674320 shutdown_connections 2024-03-25T19:31:04.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.910+0000 7fd25a515640 1 -- 172.21.15.123:0/1141674320 wait complete. 2024-03-25T19:31:04.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.911+0000 7fd25a515640 10 monclient: build_initial_monmap 2024-03-25T19:31:04.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.911+0000 7fd25a515640 10 monclient: monmap: 2024-03-25T19:31:04.912 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.911+0000 7fd25a515640 1 librados: starting msgr at 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.911+0000 7fd25a515640 1 librados: starting objecter 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.911+0000 7fd25a515640 1 Processor -- start 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.912+0000 7fd25a515640 1 -- start start 2024-03-25T19:31:04.914 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.912+0000 7fd25a515640 1 librados: setting wanted keys 2024-03-25T19:31:04.914 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.912+0000 7fd25a515640 1 librados: calling monclient init 2024-03-25T19:31:04.914 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.912+0000 7fd25a515640 10 monclient: init 2024-03-25T19:31:04.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.912+0000 7fd25a515640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:04.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.912+0000 7fd25a515640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:04.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.912+0000 7fd25a515640 10 monclient(hunting): picked mon.noname-a con 0x7fd2540a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:04.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.913+0000 7fd25a515640 10 monclient(hunting): picked mon.noname-b con 0x7fd25413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:04.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.913+0000 7fd25a515640 10 monclient(hunting): picked mon.noname-c con 0x7fd2541430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:04.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.913+0000 7fd25a515640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2540b07c0 con 0x7fd2540a98f0 2024-03-25T19:31:04.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.913+0000 7fd25a515640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2540aef00 con 0x7fd25413f710 2024-03-25T19:31:04.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.913+0000 7fd25a515640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2540b0a30 con 0x7fd2541430c0 2024-03-25T19:31:04.915 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.913+0000 7fd25a515640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:04.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.913+0000 7fd25a515640 10 monclient(hunting): authenticate will time out at 6102.273002s 2024-03-25T19:31:04.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.917+0000 7fd2537fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd25413f710 0x7fd2540b11c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:36466/0 (socket says 172.21.15.123:36466) 2024-03-25T19:31:04.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.917+0000 7fd2537fe640 1 -- 172.21.15.123:0/4019581588 learned_addr learned my addr 172.21.15.123:0/4019581588 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:04.919 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.919+0000 7fd250ff9640 10 client.?.objecter ms_handle_connect 0x7fd25413f710 2024-03-25T19:31:04.919 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.919+0000 7fd250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:04.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.922+0000 7fd250ff9640 10 client.?.objecter ms_handle_connect 0x7fd2540a98f0 2024-03-25T19:31:04.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.922+0000 7fd250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:04.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.922+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970104571 0 0) 0x7fd2540aef00 con 0x7fd25413f710 2024-03-25T19:31:04.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.922+0000 7fd250ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.922+0000 7fd250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.922+0000 7fd250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.922+0000 7fd250ff9640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:31:04.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.922+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd240004eb0 con 0x7fd25413f710 2024-03-25T19:31:04.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.927+0000 7fd250ff9640 10 client.?.objecter ms_handle_connect 0x7fd2541430c0 2024-03-25T19:31:04.926 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.927+0000 7fd250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:04.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.932+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400606258 0 0) 0x7fd2540b07c0 con 0x7fd2540a98f0 2024-03-25T19:31:04.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.932+0000 7fd250ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.932+0000 7fd250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.932+0000 7fd250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.932+0000 7fd250ff9640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:31:04.933 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.932+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2540aef00 con 0x7fd2540a98f0 2024-03-25T19:31:04.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.937+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3306209317 0 0) 0x7fd240004eb0 con 0x7fd25413f710 2024-03-25T19:31:04.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.937+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2540b07c0 con 0x7fd25413f710 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.937+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd248001960 con 0x7fd25413f710 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.937+0000 7fd250ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.937+0000 7fd250ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.937+0000 7fd250ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:04.938 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:04.939 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:04.939 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:04.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.937+0000 7fd250ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:04.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.940+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149790028 0 0) 0x7fd2540b0a30 con 0x7fd2541430c0 2024-03-25T19:31:04.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.940+0000 7fd250ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.940+0000 7fd250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.940+0000 7fd250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.940+0000 7fd250ff9640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:31:04.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.940+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd240004eb0 con 0x7fd2541430c0 2024-03-25T19:31:04.956 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 847992990 0 0) 0x7fd2540b07c0 con 0x7fd25413f710 2024-03-25T19:31:04.956 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 >> v1:172.21.15.123:6790/0 conn(0x7fd2541430c0 legacy=0x7fd2540af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.956 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 >> v1:172.21.15.123:6789/0 conn(0x7fd2540a98f0 legacy=0x7fd2540abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 1 monclient: found mon.b 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2540ae880 con 0x7fd25413f710 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 10 monclient: _finish_auth 0 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd250ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd25a515640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd25a515640 10 monclient: _renew_subs 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd25a515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd25a515640 1 -- 172.21.15.123:0/4019581588 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2540aeb30 con 0x7fd25413f710 2024-03-25T19:31:04.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd25a515640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:04.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd25a515640 10 monclient: _renew_subs 2024-03-25T19:31:04.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd25a515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:04.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd25a515640 1 -- 172.21.15.123:0/4019581588 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2541e2e40 con 0x7fd25413f710 2024-03-25T19:31:04.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.956+0000 7fd25a515640 1 librados: init done 2024-03-25T19:31:04.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.958+0000 7fd25a515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:04.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.958+0000 7fd25a515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:04.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.958+0000 7fd25a515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:04.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:04.958+0000 7fd25a515640 1 -- 172.21.15.123:0/4019581588 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2540af090 con 0x7fd25413f710 2024-03-25T19:31:05.245 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.245+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2480035f0 con 0x7fd25413f710 2024-03-25T19:31:05.245 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.245+0000 7fd250ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:05.245 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.245+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd2480037a0 con 0x7fd25413f710 2024-03-25T19:31:05.245 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.245+0000 7fd250ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:05.245 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.245+0000 7fd250ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:05.245 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.245+0000 7fd250ff9640 10 monclient: dump: 2024-03-25T19:31:05.245 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:05.246 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:05.246 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:05.246 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:05.246 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:05.246 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:05.246 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:05.246 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:05.246 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:05.246 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:05.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.245+0000 7fd2527fc640 4 set_mon_vals no callback set 2024-03-25T19:31:05.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.246+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fd2480039c0 con 0x7fd25413f710 2024-03-25T19:31:05.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.247+0000 7fd253fff640 10 monclient: get_auth_request con 0x7fd24006de00 auth_method 0 2024-03-25T19:31:05.247 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.247+0000 7fd250ff9640 10 client.4603.objecter ms_handle_connect 0x7fd24006de00 2024-03-25T19:31:05.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.251+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fd2480545b0 con 0x7fd25413f710 2024-03-25T19:31:05.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.251+0000 7fd250ff9640 10 client.4603.objecter ms_dispatch 0x7fd254000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:05.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.251+0000 7fd250ff9640 3 client.4603.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:05.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.251+0000 7fd250ff9640 3 client.4603.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:05.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.252+0000 7fd250ff9640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-25T19:31:05.262 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.263+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2480500c0 con 0x7fd25413f710 2024-03-25T19:31:05.262 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.263+0000 7fd250ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:05.263 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.263+0000 7fd250ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:05.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.509+0000 7fd25a515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:05.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.509+0000 7fd25a515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:05.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.509+0000 7fd25a515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:05.509 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.509+0000 7fd25a515640 1 -- 172.21.15.123:0/4019581588 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2540aecc0 con 0x7fd25413f710 2024-03-25T19:31:05.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.541+0000 7fd250ff9640 1 -- 172.21.15.123:0/4019581588 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fd248051ff0 con 0x7fd25413f710 2024-03-25T19:31:05.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.541+0000 7fd250ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:05.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.541+0000 7fd250ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:05.541 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:31:05.541 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.543+0000 7fd25a515640 10 librados: watch_flush enter 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.543+0000 7fd25a515640 10 librados: watch_flush exit 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.543+0000 7fd25a515640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.543+0000 7fd25a515640 10 client.4603.objecter successfully canceled tick 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.543+0000 7fd25a515640 1 -- 172.21.15.123:0/4019581588 >> v1:172.21.15.151:6800/727684239 conn(0x7fd24006de00 legacy=0x7fd2400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.543+0000 7fd25a515640 10 monclient: shutdown 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.543+0000 7fd25a515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.543+0000 7fd25a515640 1 -- 172.21.15.123:0/4019581588 >> v1:172.21.15.151:6789/0 conn(0x7fd25413f710 legacy=0x7fd2540b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.543+0000 7fd25a515640 1 -- 172.21.15.123:0/4019581588 shutdown_connections 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.543+0000 7fd25a515640 1 -- 172.21.15.123:0/4019581588 >> 172.21.15.123:0/4019581588 conn(0x7fd2540b7ea0 msgr2=0x7fd254074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:05.544 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.544+0000 7fd25a515640 1 -- 172.21.15.123:0/4019581588 shutdown_connections 2024-03-25T19:31:05.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.544+0000 7fd25a515640 1 -- 172.21.15.123:0/4019581588 wait complete. 2024-03-25T19:31:05.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:05.544+0000 7fd25a515640 1 librados: shutdown 2024-03-25T19:31:05.556 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:12.363 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:12.475 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.476+0000 7fd5edec1640 10 monclient: get_monmap_and_config 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.476+0000 7fd5edec1640 10 monclient: build_initial_monmap 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.476+0000 7fd5edec1640 10 monclient: monmap: 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:31:12.477013+0000 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:31:12.477013+0000 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:31:12.476 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.476+0000 7fd5edec1640 1 Processor -- start 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 1 -- start start 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 10 monclient: init 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e80b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e80b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e814be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e80af940 con 0x7fd5e814be90 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e80b0a70 con 0x7fd5e80b4d60 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e80b0d50 con 0x7fd5e80b1730 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:12.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.477+0000 7fd5edec1640 10 monclient(hunting): authenticate will time out at 6109.837647s 2024-03-25T19:31:12.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.482+0000 7fd5e7fff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fd5e814be90 0x7fd5e814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:40126/0 (socket says 172.21.15.123:40126) 2024-03-25T19:31:12.482 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.482+0000 7fd5e7fff640 1 -- 172.21.15.123:0/916088429 learned_addr learned my addr 172.21.15.123:0/916088429 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:12.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.497+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719971120 0 0) 0x7fd5e80b0d50 con 0x7fd5e80b1730 2024-03-25T19:31:12.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.497+0000 7fd5e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.497+0000 7fd5e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.497+0000 7fd5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.497+0000 7fd5e67fc640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:31:12.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.497+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d0003520 con 0x7fd5e80b1730 2024-03-25T19:31:12.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.500+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254144646 0 0) 0x7fd5e80af940 con 0x7fd5e814be90 2024-03-25T19:31:12.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.500+0000 7fd5e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.500 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.500+0000 7fd5e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.500 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.500+0000 7fd5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.500 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.500+0000 7fd5e67fc640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:31:12.500 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.500+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e80b0d50 con 0x7fd5e814be90 2024-03-25T19:31:12.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.511+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285211926 0 0) 0x7fd5e80b0a70 con 0x7fd5e80b4d60 2024-03-25T19:31:12.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.511+0000 7fd5e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.511+0000 7fd5e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.511+0000 7fd5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.511+0000 7fd5e67fc640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:31:12.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.511+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e80af940 con 0x7fd5e80b4d60 2024-03-25T19:31:12.512 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441685231 0 0) 0x7fd5e80b0d50 con 0x7fd5e814be90 2024-03-25T19:31:12.512 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 >> v1:172.21.15.123:6790/0 conn(0x7fd5e80b1730 legacy=0x7fd5e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 >> v1:172.21.15.151:6789/0 conn(0x7fd5e80b4d60 legacy=0x7fd5e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 1 monclient: found mon.noname-a 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e80b1030 con 0x7fd5e814be90 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd5e00019f0 con 0x7fd5e814be90 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.512+0000 7fd5e67fc640 10 monclient: dump: 2024-03-25T19:31:12.513 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.513+0000 7fd5edec1640 5 monclient: authenticate success, global_id 4596 2024-03-25T19:31:12.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.513+0000 7fd5edec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:12.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.518+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5e0001b50 con 0x7fd5e814be90 2024-03-25T19:31:12.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.518+0000 7fd5e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.518+0000 7fd5e67fc640 1 -- 172.21.15.123:0/916088429 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd5e0001f60 con 0x7fd5e814be90 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.518+0000 7fd5edec1640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.518+0000 7fd5edec1640 4 set_mon_vals no callback set 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.518+0000 7fd5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.518+0000 7fd5e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.518+0000 7fd5e67fc640 10 monclient: dump: 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:12.519 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:12.520 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:12.520 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:12.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.519+0000 7fd5edec1640 10 monclient: shutdown 2024-03-25T19:31:12.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.519+0000 7fd5edec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.519+0000 7fd5edec1640 1 -- 172.21.15.123:0/916088429 >> v1:172.21.15.123:6789/0 conn(0x7fd5e814be90 legacy=0x7fd5e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.519+0000 7fd5edec1640 1 -- 172.21.15.123:0/916088429 shutdown_connections 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.519+0000 7fd5edec1640 1 -- 172.21.15.123:0/916088429 >> 172.21.15.123:0/916088429 conn(0x7fd5e80617a0 msgr2=0x7fd5e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.519+0000 7fd5edec1640 1 -- 172.21.15.123:0/916088429 shutdown_connections 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.519+0000 7fd5edec1640 1 -- 172.21.15.123:0/916088429 wait complete. 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.520+0000 7fd5edec1640 10 monclient: build_initial_monmap 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.520+0000 7fd5edec1640 10 monclient: monmap: 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:31:12.522 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.520+0000 7fd5edec1640 1 librados: starting msgr at 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.520+0000 7fd5edec1640 1 librados: starting objecter 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.520+0000 7fd5edec1640 1 Processor -- start 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.520+0000 7fd5edec1640 1 -- start start 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.520+0000 7fd5edec1640 1 librados: setting wanted keys 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.520+0000 7fd5edec1640 1 librados: calling monclient init 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.520+0000 7fd5edec1640 10 monclient: init 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.521+0000 7fd5edec1640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.521+0000 7fd5edec1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.521+0000 7fd5edec1640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e80b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.521+0000 7fd5edec1640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e80b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:12.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.521+0000 7fd5edec1640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e814be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:12.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.521+0000 7fd5edec1640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e81d7730 con 0x7fd5e814be90 2024-03-25T19:31:12.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.521+0000 7fd5edec1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e81d85b0 con 0x7fd5e80b1730 2024-03-25T19:31:12.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.521+0000 7fd5edec1640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e81d7980 con 0x7fd5e80b4d60 2024-03-25T19:31:12.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.521+0000 7fd5edec1640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:12.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.521+0000 7fd5edec1640 10 monclient(hunting): authenticate will time out at 6109.881385s 2024-03-25T19:31:12.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.523+0000 7fd5e77fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd5e80b1730 0x7fd5e81d7ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:59984/0 (socket says 172.21.15.123:59984) 2024-03-25T19:31:12.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.523+0000 7fd5e77fe640 1 -- 172.21.15.123:0/3473791116 learned_addr learned my addr 172.21.15.123:0/3473791116 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:12.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.523+0000 7fd5cffff640 10 client.?.objecter ms_handle_connect 0x7fd5e80b1730 2024-03-25T19:31:12.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.523+0000 7fd5cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:12.530 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.530+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571914150 0 0) 0x7fd5e81d85b0 con 0x7fd5e80b1730 2024-03-25T19:31:12.530 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.530+0000 7fd5cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.530+0000 7fd5cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.530+0000 7fd5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.530+0000 7fd5cffff640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:31:12.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.530+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d4004eb0 con 0x7fd5e80b1730 2024-03-25T19:31:12.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.533+0000 7fd5cffff640 10 client.?.objecter ms_handle_connect 0x7fd5e814be90 2024-03-25T19:31:12.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.533+0000 7fd5cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:12.536 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.536+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2482206579 0 0) 0x7fd5d4004eb0 con 0x7fd5e80b1730 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.536+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5e81d85b0 con 0x7fd5e80b1730 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.537+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd5d8001a10 con 0x7fd5e80b1730 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.537+0000 7fd5cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.537+0000 7fd5cffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.537+0000 7fd5cffff640 10 monclient(hunting): dump: 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:12.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.537+0000 7fd5cffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:12.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.542+0000 7fd5cffff640 10 client.?.objecter ms_handle_connect 0x7fd5e80b4d60 2024-03-25T19:31:12.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.542+0000 7fd5cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:12.542 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.542+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311933201 0 0) 0x7fd5e81d7730 con 0x7fd5e814be90 2024-03-25T19:31:12.542 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.542+0000 7fd5cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.542 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.542+0000 7fd5cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.542 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.542+0000 7fd5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.542 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.542+0000 7fd5cffff640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:31:12.543 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.542+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d4004eb0 con 0x7fd5e814be90 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4101841996 0 0) 0x7fd5e81d85b0 con 0x7fd5e80b1730 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 >> v1:172.21.15.123:6790/0 conn(0x7fd5e80b4d60 legacy=0x7fd5e813a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 >> v1:172.21.15.123:6789/0 conn(0x7fd5e814be90 legacy=0x7fd5e81e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 1 monclient: found mon.b 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e81394d0 con 0x7fd5e80b1730 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 10 monclient: _finish_auth 0 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:12.545 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:12.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5edec1640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:31:12.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.544+0000 7fd5edec1640 10 monclient: _renew_subs 2024-03-25T19:31:12.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:12.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 1 -- 172.21.15.123:0/3473791116 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e813a7d0 con 0x7fd5e80b1730 2024-03-25T19:31:12.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:12.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 10 monclient: _renew_subs 2024-03-25T19:31:12.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:12.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 1 -- 172.21.15.123:0/3473791116 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e813d2d0 con 0x7fd5e80b1730 2024-03-25T19:31:12.546 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 1 librados: init done 2024-03-25T19:31:12.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:12.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:12.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:12.547 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.545+0000 7fd5edec1640 1 -- 172.21.15.123:0/3473791116 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e81d7b10 con 0x7fd5e80b1730 2024-03-25T19:31:12.822 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.822+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5d8001cf0 con 0x7fd5e80b1730 2024-03-25T19:31:12.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.822+0000 7fd5cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:12.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.822+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd5d8003a60 con 0x7fd5e80b1730 2024-03-25T19:31:12.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.822+0000 7fd5cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.822+0000 7fd5cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:12.823 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.822+0000 7fd5cffff640 10 monclient: dump: 2024-03-25T19:31:12.823 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:12.823 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:12.823 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:12.823 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:12.823 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.824 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:12.824 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:12.824 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:12.824 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:12.824 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:12.824 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.822+0000 7fd5e5ffb640 4 set_mon_vals no callback set 2024-03-25T19:31:12.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.823+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fd5d8003c40 con 0x7fd5e80b1730 2024-03-25T19:31:12.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.824+0000 7fd5e6ffd640 10 monclient: get_auth_request con 0x7fd5d406dd40 auth_method 0 2024-03-25T19:31:12.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.824+0000 7fd5cffff640 10 client.4609.objecter ms_handle_connect 0x7fd5d406dd40 2024-03-25T19:31:12.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.829+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fd5d8054900 con 0x7fd5e80b1730 2024-03-25T19:31:12.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.829+0000 7fd5cffff640 10 client.4609.objecter ms_dispatch 0x7fd5e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:12.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.829+0000 7fd5cffff640 3 client.4609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:12.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.829+0000 7fd5cffff640 3 client.4609.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:12.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.829+0000 7fd5cffff640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-25T19:31:12.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.831+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5d8050240 con 0x7fd5e80b1730 2024-03-25T19:31:12.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.831+0000 7fd5cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:12.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:12.831+0000 7fd5cffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:13.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.052+0000 7fd5edec1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:13.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.052+0000 7fd5edec1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:13.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.052+0000 7fd5edec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:13.052 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.052+0000 7fd5edec1640 1 -- 172.21.15.123:0/3473791116 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5e80627b0 con 0x7fd5e80b1730 2024-03-25T19:31:13.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.178+0000 7fd5cffff640 1 -- 172.21.15.123:0/3473791116 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fd5d8052340 con 0x7fd5e80b1730 2024-03-25T19:31:13.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.178+0000 7fd5cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:13.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.178+0000 7fd5cffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:13.178 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:31:13.178 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:13.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 10 librados: watch_flush enter 2024-03-25T19:31:13.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 10 librados: watch_flush exit 2024-03-25T19:31:13.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-25T19:31:13.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 10 client.4609.objecter successfully canceled tick 2024-03-25T19:31:13.180 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 1 -- 172.21.15.123:0/3473791116 >> v1:172.21.15.151:6800/727684239 conn(0x7fd5d406dd40 legacy=0x7fd5d4070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 10 monclient: shutdown 2024-03-25T19:31:13.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:13.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 1 -- 172.21.15.123:0/3473791116 >> v1:172.21.15.151:6789/0 conn(0x7fd5e80b1730 legacy=0x7fd5e81d7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 1 -- 172.21.15.123:0/3473791116 shutdown_connections 2024-03-25T19:31:13.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 1 -- 172.21.15.123:0/3473791116 >> 172.21.15.123:0/3473791116 conn(0x7fd5e80617a0 msgr2=0x7fd5e813c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:13.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 1 -- 172.21.15.123:0/3473791116 shutdown_connections 2024-03-25T19:31:13.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 1 -- 172.21.15.123:0/3473791116 wait complete. 2024-03-25T19:31:13.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:13.180+0000 7fd5edec1640 1 librados: shutdown 2024-03-25T19:31:13.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:16.386 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:09:05.958 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:16.392 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:09:05.964 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:16.423 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:09:05.995 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:16.430 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:09:06.002 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:16.440 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:09:05.973 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:16.444 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:09:06.017 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:16.447 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:09:05.980 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:16.475 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:09:06.008 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:16.480 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:09:06.053 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:19.999 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:20.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.116+0000 7f828494b640 10 monclient: get_monmap_and_config 2024-03-25T19:31:20.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.116+0000 7f828494b640 10 monclient: build_initial_monmap 2024-03-25T19:31:20.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.116+0000 7f828494b640 10 monclient: monmap: 2024-03-25T19:31:20.117 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:20.117 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:20.117 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:31:20.117578+0000 2024-03-25T19:31:20.117 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:31:20.117578+0000 2024-03-25T19:31:20.117 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:20.117 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:20.117 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:20.117 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:31:20.117 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:31:20.117 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:20.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.117+0000 7f828494b640 1 Processor -- start 2024-03-25T19:31:20.118 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.117+0000 7f828494b640 1 -- start start 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.117+0000 7f828494b640 10 monclient: init 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.118+0000 7f828494b640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.118+0000 7f828494b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.118+0000 7f828494b640 10 monclient(hunting): picked mon.noname-c con 0x7f82800b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.118+0000 7f828494b640 10 monclient(hunting): picked mon.noname-a con 0x7f82800b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.118+0000 7f828494b640 10 monclient(hunting): picked mon.noname-b con 0x7f828014be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.118+0000 7f828494b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82800af940 con 0x7f82800b1730 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.118+0000 7f828494b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82800b0a70 con 0x7f82800b4d60 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.118+0000 7f828494b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82800b0d50 con 0x7f828014be90 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.118+0000 7f828494b640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:20.119 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.118+0000 7f828494b640 10 monclient(hunting): authenticate will time out at 6117.478332s 2024-03-25T19:31:20.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.123+0000 7f827dd9b640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f82800b1730 0x7f82800b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:40996/0 (socket says 172.21.15.123:40996) 2024-03-25T19:31:20.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.123+0000 7f827dd9b640 1 -- 172.21.15.123:0/1023922107 learned_addr learned my addr 172.21.15.123:0/1023922107 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:20.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.128+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000454368 0 0) 0x7f82800af940 con 0x7f82800b1730 2024-03-25T19:31:20.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.128+0000 7f827d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.128+0000 7f827d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.128+0000 7f827d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.128+0000 7f827d59a640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:31:20.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.128+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8268003520 con 0x7f82800b1730 2024-03-25T19:31:20.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.138+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679290279 0 0) 0x7f82800b0d50 con 0x7f828014be90 2024-03-25T19:31:20.138 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.138+0000 7f827d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.138+0000 7f827d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.138+0000 7f827d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.138+0000 7f827d59a640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:31:20.139 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.138+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82800af940 con 0x7f828014be90 2024-03-25T19:31:20.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3139960115 0 0) 0x7f8268003520 con 0x7f82800b1730 2024-03-25T19:31:20.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 >> v1:172.21.15.123:6790/0 conn(0x7f828014be90 legacy=0x7f828014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 >> v1:172.21.15.151:6789/0 conn(0x7f82800b4d60 legacy=0x7f828013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:31:20.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 1 monclient: found mon.noname-a 2024-03-25T19:31:20.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:20.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:31:20.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82800b1030 con 0x7f82800b1730 2024-03-25T19:31:20.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 10 monclient: _finish_auth 0 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8274001690 con 0x7f82800b1730 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f828494b640 5 monclient: authenticate success, global_id 4602 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f828494b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f827d59a640 10 monclient: dump: 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:20.144 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:20.145 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:20.145 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:20.145 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:20.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.143+0000 7f828494b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:20.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.153+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82740017f0 con 0x7f82800b1730 2024-03-25T19:31:20.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.153+0000 7f827d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:20.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.153+0000 7f827d59a640 1 -- 172.21.15.123:0/1023922107 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8274001c00 con 0x7f82800b1730 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.153+0000 7f827d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.153+0000 7f827d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.153+0000 7f827d59a640 10 monclient: dump: 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:20.154 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:20.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.153+0000 7f828494b640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:20.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.153+0000 7f828494b640 4 set_mon_vals no callback set 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.154+0000 7f828494b640 10 monclient: shutdown 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.154+0000 7f828494b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.154+0000 7f828494b640 1 -- 172.21.15.123:0/1023922107 >> v1:172.21.15.123:6789/0 conn(0x7f82800b1730 legacy=0x7f82800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.154+0000 7f828494b640 1 -- 172.21.15.123:0/1023922107 shutdown_connections 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.154+0000 7f828494b640 1 -- 172.21.15.123:0/1023922107 >> 172.21.15.123:0/1023922107 conn(0x7f82800617a0 msgr2=0x7f8280061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.154+0000 7f828494b640 1 -- 172.21.15.123:0/1023922107 shutdown_connections 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.154+0000 7f828494b640 1 -- 172.21.15.123:0/1023922107 wait complete. 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.155+0000 7f828494b640 10 monclient: build_initial_monmap 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.155+0000 7f828494b640 10 monclient: monmap: 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:31:20.157 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 1 librados: starting msgr at 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 1 librados: starting objecter 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 1 Processor -- start 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 1 -- start start 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 1 librados: setting wanted keys 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 1 librados: calling monclient init 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 10 monclient: init 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:20.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:20.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 10 monclient(hunting): picked mon.noname-a con 0x7f82800b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:20.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.156+0000 7f828494b640 10 monclient(hunting): picked mon.noname-b con 0x7f82800b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:20.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.157+0000 7f828494b640 10 monclient(hunting): picked mon.noname-c con 0x7f828014be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:20.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.157+0000 7f828494b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8280144d40 con 0x7f82800b1730 2024-03-25T19:31:20.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.157+0000 7f828494b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82801dbac0 con 0x7f82800b4d60 2024-03-25T19:31:20.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.157+0000 7f828494b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8280135d30 con 0x7f828014be90 2024-03-25T19:31:20.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.157+0000 7f828494b640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:20.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.157+0000 7f828494b640 10 monclient(hunting): authenticate will time out at 6117.517001s 2024-03-25T19:31:20.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.164+0000 7f827dd9b640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f82800b4d60 0x7f8280144f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:57770/0 (socket says 172.21.15.123:57770) 2024-03-25T19:31:20.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.164+0000 7f827dd9b640 1 -- 172.21.15.123:0/634659420 learned_addr learned my addr 172.21.15.123:0/634659420 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:20.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.165+0000 7f826effd640 10 client.?.objecter ms_handle_connect 0x7f82800b4d60 2024-03-25T19:31:20.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.165+0000 7f826effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:20.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.166+0000 7f826effd640 10 client.?.objecter ms_handle_connect 0x7f828014be90 2024-03-25T19:31:20.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.166+0000 7f826effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:20.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.172+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996430237 0 0) 0x7f82801dbac0 con 0x7f82800b4d60 2024-03-25T19:31:20.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.173+0000 7f826effd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.173+0000 7f826effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.173+0000 7f826effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.173+0000 7f826effd640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:31:20.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.173+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8260004eb0 con 0x7f82800b4d60 2024-03-25T19:31:20.176 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.176+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583561927 0 0) 0x7f8280135d30 con 0x7f828014be90 2024-03-25T19:31:20.176 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.176+0000 7f826effd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.176 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.176+0000 7f826effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.176 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.176+0000 7f826effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.176 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.176+0000 7f826effd640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:31:20.176 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.176+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82801dbac0 con 0x7f828014be90 2024-03-25T19:31:20.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.184+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2158020412 0 0) 0x7f8260004eb0 con 0x7f82800b4d60 2024-03-25T19:31:20.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.184+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8280135d30 con 0x7f82800b4d60 2024-03-25T19:31:20.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.184+0000 7f826effd640 10 client.?.objecter ms_handle_connect 0x7f82800b1730 2024-03-25T19:31:20.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.184+0000 7f826effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:20.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.184+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8274001bc0 con 0x7f82800b4d60 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.184+0000 7f826effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.184+0000 7f826effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.184+0000 7f826effd640 10 monclient(hunting): dump: 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:20.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.184+0000 7f826effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:20.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.191+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611579619 0 0) 0x7f8280144d40 con 0x7f82800b1730 2024-03-25T19:31:20.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.191+0000 7f826effd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.191+0000 7f826effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.191+0000 7f826effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.191+0000 7f826effd640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:31:20.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.191+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8260005870 con 0x7f82800b1730 2024-03-25T19:31:20.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 344644928 0 0) 0x7f8280135d30 con 0x7f82800b4d60 2024-03-25T19:31:20.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 >> v1:172.21.15.123:6790/0 conn(0x7f828014be90 legacy=0x7f82801e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 >> v1:172.21.15.123:6789/0 conn(0x7f82800b1730 legacy=0x7f82801db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 1 monclient: found mon.b 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82801382f0 con 0x7f82800b4d60 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 10 monclient: _finish_auth 0 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f826effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f828494b640 5 monclient: authenticate success, global_id 4615 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f828494b640 10 monclient: _renew_subs 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f828494b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f828494b640 1 -- 172.21.15.123:0/634659420 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8280145670 con 0x7f82800b4d60 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f828494b640 10 client.4615.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f828494b640 10 monclient: _renew_subs 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f828494b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:20.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f828494b640 1 -- 172.21.15.123:0/634659420 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82801447c0 con 0x7f82800b4d60 2024-03-25T19:31:20.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.195+0000 7f828494b640 1 librados: init done 2024-03-25T19:31:20.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.196+0000 7f828494b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:20.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.196+0000 7f828494b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:20.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.196+0000 7f828494b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:20.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.196+0000 7f828494b640 1 -- 172.21.15.123:0/634659420 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82800627b0 con 0x7f82800b4d60 2024-03-25T19:31:20.495 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.495+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82740011f0 con 0x7f82800b4d60 2024-03-25T19:31:20.495 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.495+0000 7f826effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.495+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8274001bc0 con 0x7f82800b4d60 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.495+0000 7f826effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.495+0000 7f826effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.495+0000 7f826effd640 10 monclient: dump: 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:20.496 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:20.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.495+0000 7f827cd99640 4 set_mon_vals no callback set 2024-03-25T19:31:20.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.496+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f827401a980 con 0x7f82800b4d60 2024-03-25T19:31:20.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.497+0000 7f827e59c640 10 monclient: get_auth_request con 0x7f826006e160 auth_method 0 2024-03-25T19:31:20.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.497+0000 7f826effd640 10 client.4615.objecter ms_handle_connect 0x7f826006e160 2024-03-25T19:31:20.501 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.502+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f8274054630 con 0x7f82800b4d60 2024-03-25T19:31:20.501 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.502+0000 7f826effd640 10 client.4615.objecter ms_dispatch 0x7f8280000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:20.502 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.502+0000 7f826effd640 3 client.4615.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:20.502 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.502+0000 7f826effd640 3 client.4615.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:20.502 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.502+0000 7f826effd640 20 client.4615.objecter dump_active .. 0 homeless 2024-03-25T19:31:20.502 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.502+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f827404ff70 con 0x7f82800b4d60 2024-03-25T19:31:20.502 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.502+0000 7f826effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:20.502 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.502+0000 7f826effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:20.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.761+0000 7f828494b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:20.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.761+0000 7f828494b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:20.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.761+0000 7f828494b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:20.761 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.761+0000 7f828494b640 1 -- 172.21.15.123:0/634659420 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8280069f40 con 0x7f82800b4d60 2024-03-25T19:31:20.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.791+0000 7f826effd640 1 -- 172.21.15.123:0/634659420 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f8274052070 con 0x7f82800b4d60 2024-03-25T19:31:20.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.791+0000 7f826effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:20.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.791+0000 7f826effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:20.791 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:31:20.791 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:20.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.793+0000 7f828494b640 10 librados: watch_flush enter 2024-03-25T19:31:20.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 10 librados: watch_flush exit 2024-03-25T19:31:20.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 20 client.4615.objecter shutdown clearing up homeless session... 2024-03-25T19:31:20.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 10 client.4615.objecter successfully canceled tick 2024-03-25T19:31:20.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 1 -- 172.21.15.123:0/634659420 >> v1:172.21.15.151:6800/727684239 conn(0x7f826006e160 legacy=0x7f8260070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 10 monclient: shutdown 2024-03-25T19:31:20.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:20.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 1 -- 172.21.15.123:0/634659420 >> v1:172.21.15.151:6789/0 conn(0x7f82800b4d60 legacy=0x7f8280144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 1 -- 172.21.15.123:0/634659420 shutdown_connections 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 1 -- 172.21.15.123:0/634659420 >> 172.21.15.123:0/634659420 conn(0x7f82800617a0 msgr2=0x7f82800b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 1 -- 172.21.15.123:0/634659420 shutdown_connections 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 1 -- 172.21.15.123:0/634659420 wait complete. 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:20.794+0000 7f828494b640 1 librados: shutdown 2024-03-25T19:31:20.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:27.612 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:27.722 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.723+0000 7f22bfcb3640 10 monclient: get_monmap_and_config 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.723+0000 7f22bfcb3640 10 monclient: build_initial_monmap 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.723+0000 7f22bfcb3640 10 monclient: monmap: 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:31:27.724177+0000 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:31:27.724177+0000 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:31:27.723 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.723+0000 7f22bfcb3640 1 Processor -- start 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 1 -- start start 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 10 monclient: init 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 10 monclient(hunting): picked mon.noname-a con 0x7f22b80a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 10 monclient(hunting): picked mon.noname-c con 0x7f22b813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 10 monclient(hunting): picked mon.noname-b con 0x7f22b81430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b8142520 con 0x7f22b80a98f0 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b80b28a0 con 0x7f22b813f710 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b80b2a30 con 0x7f22b81430c0 2024-03-25T19:31:27.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:27.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.724+0000 7f22bfcb3640 10 monclient(hunting): authenticate will time out at 6125.084740s 2024-03-25T19:31:27.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.730+0000 7f22bda4f640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f22b80a98f0 0x7f22b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:54944/0 (socket says 172.21.15.123:54944) 2024-03-25T19:31:27.730 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.730+0000 7f22bda4f640 1 -- 172.21.15.123:0/2781703787 learned_addr learned my addr 172.21.15.123:0/2781703787 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:27.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.741+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676768841 0 0) 0x7f22b80b28a0 con 0x7f22b813f710 2024-03-25T19:31:27.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.741+0000 7f22bca4d640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.741+0000 7f22bca4d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.741+0000 7f22bca4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.741+0000 7f22bca4d640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:31:27.741 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.741+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a4003520 con 0x7f22b813f710 2024-03-25T19:31:27.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.742+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608786576 0 0) 0x7f22b8142520 con 0x7f22b80a98f0 2024-03-25T19:31:27.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.742+0000 7f22bca4d640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.742+0000 7f22bca4d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.742+0000 7f22bca4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.742+0000 7f22bca4d640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:31:27.743 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.742+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b80b28a0 con 0x7f22b80a98f0 2024-03-25T19:31:27.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481976183 0 0) 0x7f22b80b28a0 con 0x7f22b80a98f0 2024-03-25T19:31:27.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 >> v1:172.21.15.123:6790/0 conn(0x7f22b81430c0 legacy=0x7f22b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:27.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 >> v1:172.21.15.151:6789/0 conn(0x7f22b813f710 legacy=0x7f22b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:27.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 10 monclient: _finish_hunting 0 2024-03-25T19:31:27.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 1 monclient: found mon.noname-a 2024-03-25T19:31:27.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:27.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b80b44c0 con 0x7f22b80a98f0 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 10 monclient: _finish_auth 0 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 10 monclient: _check_auth_tickets 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f22b4001670 con 0x7f22b80a98f0 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bfcb3640 5 monclient: authenticate success, global_id 4608 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bfcb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bca4d640 10 monclient: dump: 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:27.757 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:27.758 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:27.758 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:27.758 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:27.758 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:27.758 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.756+0000 7f22bfcb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:27.786 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22b40017d0 con 0x7f22b80a98f0 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bca4d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bca4d640 1 -- 172.21.15.123:0/2781703787 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f22b4001ba0 con 0x7f22b80a98f0 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bca4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bca4d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bca4d640 10 monclient: dump: 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:27.787 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:27.788 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:27.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bfcb3640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:27.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bfcb3640 4 set_mon_vals no callback set 2024-03-25T19:31:27.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bfcb3640 10 monclient: shutdown 2024-03-25T19:31:27.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bfcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:27.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.787+0000 7f22bfcb3640 1 -- 172.21.15.123:0/2781703787 >> v1:172.21.15.123:6789/0 conn(0x7f22b80a98f0 legacy=0x7f22b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:27.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.788+0000 7f22bfcb3640 1 -- 172.21.15.123:0/2781703787 shutdown_connections 2024-03-25T19:31:27.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.788+0000 7f22bfcb3640 1 -- 172.21.15.123:0/2781703787 >> 172.21.15.123:0/2781703787 conn(0x7f22b80b7ea0 msgr2=0x7f22b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:27.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.788+0000 7f22bfcb3640 1 -- 172.21.15.123:0/2781703787 shutdown_connections 2024-03-25T19:31:27.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.788+0000 7f22bfcb3640 1 -- 172.21.15.123:0/2781703787 wait complete. 2024-03-25T19:31:27.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.789+0000 7f22bfcb3640 10 monclient: build_initial_monmap 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.789+0000 7f22bfcb3640 10 monclient: monmap: 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.789+0000 7f22bfcb3640 1 librados: starting msgr at 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.789+0000 7f22bfcb3640 1 librados: starting objecter 2024-03-25T19:31:27.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.789+0000 7f22bfcb3640 1 Processor -- start 2024-03-25T19:31:27.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.789+0000 7f22bfcb3640 1 -- start start 2024-03-25T19:31:27.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.789+0000 7f22bfcb3640 1 librados: setting wanted keys 2024-03-25T19:31:27.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.789+0000 7f22bfcb3640 1 librados: calling monclient init 2024-03-25T19:31:27.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.789+0000 7f22bfcb3640 10 monclient: init 2024-03-25T19:31:27.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.790+0000 7f22bfcb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:27.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.790+0000 7f22bfcb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:27.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.790+0000 7f22bfcb3640 10 monclient(hunting): picked mon.noname-c con 0x7f22b80a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:27.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.790+0000 7f22bfcb3640 10 monclient(hunting): picked mon.noname-b con 0x7f22b813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:27.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.790+0000 7f22bfcb3640 10 monclient(hunting): picked mon.noname-a con 0x7f22b81430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:27.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.790+0000 7f22bfcb3640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b81da8f0 con 0x7f22b81430c0 2024-03-25T19:31:27.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.790+0000 7f22bfcb3640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b81399f0 con 0x7f22b813f710 2024-03-25T19:31:27.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.790+0000 7f22bfcb3640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b81db770 con 0x7f22b80a98f0 2024-03-25T19:31:27.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.790+0000 7f22bfcb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:27.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.790+0000 7f22bfcb3640 10 monclient(hunting): authenticate will time out at 6125.150455s 2024-03-25T19:31:27.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.795+0000 7f22bd24e640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f22b813f710 0x7f22b81381f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:51770/0 (socket says 172.21.15.123:51770) 2024-03-25T19:31:27.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.795+0000 7f22bd24e640 1 -- 172.21.15.123:0/4065614428 learned_addr learned my addr 172.21.15.123:0/4065614428 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:27.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.797+0000 7f22ae7fc640 10 client.?.objecter ms_handle_connect 0x7f22b813f710 2024-03-25T19:31:27.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.797+0000 7f22ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:27.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.807+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259055158 0 0) 0x7f22b81399f0 con 0x7f22b813f710 2024-03-25T19:31:27.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.807+0000 7f22ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.807+0000 7f22ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.807+0000 7f22ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.807+0000 7f22ae7fc640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:31:27.807 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.807+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f229c004eb0 con 0x7f22b813f710 2024-03-25T19:31:27.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.808+0000 7f22ae7fc640 10 client.?.objecter ms_handle_connect 0x7f22b80a98f0 2024-03-25T19:31:27.808 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.808+0000 7f22ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:27.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.818+0000 7f22ae7fc640 10 client.?.objecter ms_handle_connect 0x7f22b81430c0 2024-03-25T19:31:27.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.818+0000 7f22ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:27.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.828+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021167272 0 0) 0x7f22b81db770 con 0x7f22b80a98f0 2024-03-25T19:31:27.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.828+0000 7f22ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.828+0000 7f22ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.828+0000 7f22ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.828+0000 7f22ae7fc640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:31:27.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.828+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b81399f0 con 0x7f22b80a98f0 2024-03-25T19:31:27.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.832+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3773985278 0 0) 0x7f229c004eb0 con 0x7f22b813f710 2024-03-25T19:31:27.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.832+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22b81db770 con 0x7f22b813f710 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.832+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f22a8001990 con 0x7f22b813f710 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.832+0000 7f22ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.832+0000 7f22ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.832+0000 7f22ae7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:27.833 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:27.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.832+0000 7f22ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:27.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.833+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548667748 0 0) 0x7f22b81da8f0 con 0x7f22b81430c0 2024-03-25T19:31:27.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.833+0000 7f22ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.833+0000 7f22ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.833+0000 7f22ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.833+0000 7f22ae7fc640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:31:27.834 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.833+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f229c004eb0 con 0x7f22b81430c0 2024-03-25T19:31:27.849 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.849+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3329582601 0 0) 0x7f229c004eb0 con 0x7f22b81430c0 2024-03-25T19:31:27.849 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.849+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22b81da8f0 con 0x7f22b81430c0 2024-03-25T19:31:27.849 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.849+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f22b0001a90 con 0x7f22b81430c0 2024-03-25T19:31:27.849 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.849+0000 7f22ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:27.849 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.849+0000 7f22ae7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.849+0000 7f22ae7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:27.850 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:27.934 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2073609690 0 0) 0x7f22b81da8f0 con 0x7f22b81430c0 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 >> v1:172.21.15.123:6790/0 conn(0x7f22b80a98f0 legacy=0x7f22b81db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 >> v1:172.21.15.151:6789/0 conn(0x7f22b813f710 legacy=0x7f22b81381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 1 monclient: found mon.a 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b81dab40 con 0x7f22b81430c0 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22bfcb3640 5 monclient: authenticate success, global_id 4611 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22bfcb3640 10 monclient: _renew_subs 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22bfcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:31:27.935 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22bfcb3640 1 -- 172.21.15.123:0/4065614428 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b8137870 con 0x7f22b81430c0 2024-03-25T19:31:27.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22bfcb3640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:27.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22bfcb3640 10 monclient: _renew_subs 2024-03-25T19:31:27.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22bfcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:31:27.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22bfcb3640 1 -- 172.21.15.123:0/4065614428 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b8138ae0 con 0x7f22b81430c0 2024-03-25T19:31:27.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.934+0000 7f22bfcb3640 1 librados: init done 2024-03-25T19:31:27.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.935+0000 7f22bfcb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:27.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.935+0000 7f22bfcb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:27.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.935+0000 7f22bfcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:31:27.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:27.935+0000 7f22bfcb3640 1 -- 172.21.15.123:0/4065614428 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b80627b0 con 0x7f22b81430c0 2024-03-25T19:31:28.257 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.257+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22b0001fa0 con 0x7f22b81430c0 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.257+0000 7f22ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.257+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f22b0003b10 con 0x7f22b81430c0 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.257+0000 7f22ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.257+0000 7f22ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.257+0000 7f22ae7fc640 10 monclient: dump: 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:28.258 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:28.259 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:28.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.257+0000 7f22affff640 4 set_mon_vals no callback set 2024-03-25T19:31:28.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.258+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f22b0003d30 con 0x7f22b81430c0 2024-03-25T19:31:28.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.259+0000 7f22bda4f640 10 monclient: get_auth_request con 0x7f229c06df10 auth_method 0 2024-03-25T19:31:28.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.259+0000 7f22ae7fc640 10 client.4611.objecter ms_handle_connect 0x7f229c06df10 2024-03-25T19:31:28.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.266+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f22b00537a0 con 0x7f22b81430c0 2024-03-25T19:31:28.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.266+0000 7f22ae7fc640 10 client.4611.objecter ms_dispatch 0x7f22b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:28.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.266+0000 7f22ae7fc640 3 client.4611.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:28.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.266+0000 7f22ae7fc640 3 client.4611.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:28.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.266+0000 7f22ae7fc640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-25T19:31:28.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.266+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22b004f2b0 con 0x7f22b81430c0 2024-03-25T19:31:28.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.266+0000 7f22ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:28.267 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.266+0000 7f22ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:28.517 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.517+0000 7f22bfcb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:28.517 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.517+0000 7f22bfcb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:28.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.517+0000 7f22bfcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:31:28.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.517+0000 7f22bfcb3640 1 -- 172.21.15.123:0/4065614428 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22b8069f40 con 0x7f22b81430c0 2024-03-25T19:31:28.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.549+0000 7f22ae7fc640 1 -- 172.21.15.123:0/4065614428 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f22b00511e0 con 0x7f22b81430c0 2024-03-25T19:31:28.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.549+0000 7f22ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:28.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.549+0000 7f22ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:28.550 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:31:28.550 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:28.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 10 librados: watch_flush enter 2024-03-25T19:31:28.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 10 librados: watch_flush exit 2024-03-25T19:31:28.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-25T19:31:28.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 10 client.4611.objecter successfully canceled tick 2024-03-25T19:31:28.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 1 -- 172.21.15.123:0/4065614428 >> v1:172.21.15.151:6800/727684239 conn(0x7f229c06df10 legacy=0x7f229c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 10 monclient: shutdown 2024-03-25T19:31:28.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:28.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 1 -- 172.21.15.123:0/4065614428 >> v1:172.21.15.123:6789/0 conn(0x7f22b81430c0 legacy=0x7f22b81e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 1 -- 172.21.15.123:0/4065614428 shutdown_connections 2024-03-25T19:31:28.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 1 -- 172.21.15.123:0/4065614428 >> 172.21.15.123:0/4065614428 conn(0x7f22b80b7ea0 msgr2=0x7f22b813d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:28.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 1 -- 172.21.15.123:0/4065614428 shutdown_connections 2024-03-25T19:31:28.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 1 -- 172.21.15.123:0/4065614428 wait complete. 2024-03-25T19:31:28.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:28.552+0000 7f22bfcb3640 1 librados: shutdown 2024-03-25T19:31:28.564 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:35.370 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:35.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.487+0000 7f4a60e16640 10 monclient: get_monmap_and_config 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.487+0000 7f4a60e16640 10 monclient: build_initial_monmap 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.487+0000 7f4a60e16640 10 monclient: monmap: 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:31:35.489039+0000 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:31:35.489039+0000 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:31:35.488 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:35.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.487+0000 7f4a60e16640 1 Processor -- start 2024-03-25T19:31:35.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 1 -- start start 2024-03-25T19:31:35.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 10 monclient: init 2024-03-25T19:31:35.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:35.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:35.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 10 monclient(hunting): picked mon.noname-c con 0x7f4a5c0b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:35.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 10 monclient(hunting): picked mon.noname-a con 0x7f4a5c0b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:35.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5c14be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:35.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c0af940 con 0x7f4a5c0b1730 2024-03-25T19:31:35.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c0b0a70 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c0b0d50 con 0x7f4a5c14be90 2024-03-25T19:31:35.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:35.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.488+0000 7f4a60e16640 10 monclient(hunting): authenticate will time out at 6132.849664s 2024-03-25T19:31:35.502 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.501+0000 7f4a53fff640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4a5c0b1730 0x7f4a5c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:51448/0 (socket says 172.21.15.123:51448) 2024-03-25T19:31:35.502 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.501+0000 7f4a53fff640 1 -- 172.21.15.123:0/3510402620 learned_addr learned my addr 172.21.15.123:0/3510402620 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:35.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.520+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605884689 0 0) 0x7f4a5c0b0a70 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.520+0000 7f4a59d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:31:35.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.520+0000 7f4a59d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:35.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.520+0000 7f4a59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:35.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.520+0000 7f4a59d9b640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:31:35.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.520+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a48003520 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.525+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060669797 0 0) 0x7f4a5c0af940 con 0x7f4a5c0b1730 2024-03-25T19:31:35.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.525+0000 7f4a59d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:31:35.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.525+0000 7f4a59d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:35.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.525+0000 7f4a59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:35.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.525+0000 7f4a59d9b640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:31:35.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.525+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a5c0b0a70 con 0x7f4a5c0b1730 2024-03-25T19:31:35.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.532+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203482187 0 0) 0x7f4a5c0b0d50 con 0x7f4a5c14be90 2024-03-25T19:31:35.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.532+0000 7f4a59d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:31:35.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.532+0000 7f4a59d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:35.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.532+0000 7f4a59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:35.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.532+0000 7f4a59d9b640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:31:35.533 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.532+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a5c0af940 con 0x7f4a5c14be90 2024-03-25T19:31:35.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3395478493 0 0) 0x7f4a48003520 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 >> v1:172.21.15.123:6790/0 conn(0x7f4a5c14be90 legacy=0x7f4a5c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 >> v1:172.21.15.123:6789/0 conn(0x7f4a5c0b1730 legacy=0x7f4a5c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 1 monclient: found mon.noname-c 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c0b1030 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 10 monclient: _finish_auth 0 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4a4c001690 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a60e16640 5 monclient: authenticate success, global_id 4624 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a60e16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a59d9b640 10 monclient: dump: 2024-03-25T19:31:35.539 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:35.540 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:35.540 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:35.540 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:35.540 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:35.540 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:35.540 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:35.540 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:35.540 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:35.540 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:35.540 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.537+0000 7f4a60e16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:35.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.567+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a4c0017f0 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.567+0000 7f4a59d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:35.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.567+0000 7f4a59d9b640 1 -- 172.21.15.123:0/3510402620 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4a4c001c00 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.567+0000 7f4a59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:35.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.567+0000 7f4a59d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:35.568 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.567+0000 7f4a59d9b640 10 monclient: dump: 2024-03-25T19:31:35.568 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:35.568 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:35.569 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:35.569 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:35.569 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:35.569 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:35.569 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:35.569 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:35.569 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:35.569 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:35.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.567+0000 7f4a60e16640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:35.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.567+0000 7f4a60e16640 4 set_mon_vals no callback set 2024-03-25T19:31:35.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.568+0000 7f4a60e16640 10 monclient: shutdown 2024-03-25T19:31:35.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.568+0000 7f4a60e16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:35.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.568+0000 7f4a60e16640 1 -- 172.21.15.123:0/3510402620 >> v1:172.21.15.151:6789/0 conn(0x7f4a5c0b4d60 legacy=0x7f4a5c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.568+0000 7f4a60e16640 1 -- 172.21.15.123:0/3510402620 shutdown_connections 2024-03-25T19:31:35.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.568+0000 7f4a60e16640 1 -- 172.21.15.123:0/3510402620 >> 172.21.15.123:0/3510402620 conn(0x7f4a5c0617a0 msgr2=0x7f4a5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:35.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.569+0000 7f4a60e16640 1 -- 172.21.15.123:0/3510402620 shutdown_connections 2024-03-25T19:31:35.570 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.569+0000 7f4a60e16640 1 -- 172.21.15.123:0/3510402620 wait complete. 2024-03-25T19:31:35.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.570+0000 7f4a60e16640 10 monclient: build_initial_monmap 2024-03-25T19:31:35.571 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.570+0000 7f4a60e16640 10 monclient: monmap: 2024-03-25T19:31:35.571 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:35.571 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:35.571 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:31:35.571 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.570+0000 7f4a60e16640 1 librados: starting msgr at 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.570+0000 7f4a60e16640 1 librados: starting objecter 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.570+0000 7f4a60e16640 1 Processor -- start 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.570+0000 7f4a60e16640 1 -- start start 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.570+0000 7f4a60e16640 1 librados: setting wanted keys 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.570+0000 7f4a60e16640 1 librados: calling monclient init 2024-03-25T19:31:35.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.570+0000 7f4a60e16640 10 monclient: init 2024-03-25T19:31:35.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.571+0000 7f4a60e16640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:35.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.571+0000 7f4a60e16640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:35.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.571+0000 7f4a60e16640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5c0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:35.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.571+0000 7f4a60e16640 10 monclient(hunting): picked mon.noname-c con 0x7f4a5c0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:35.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.571+0000 7f4a60e16640 10 monclient(hunting): picked mon.noname-a con 0x7f4a5c14be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:35.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.571+0000 7f4a60e16640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c142050 con 0x7f4a5c14be90 2024-03-25T19:31:35.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.572+0000 7f4a60e16640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c1dac50 con 0x7f4a5c0b1730 2024-03-25T19:31:35.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.572+0000 7f4a60e16640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c140d00 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.572+0000 7f4a60e16640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:35.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.572+0000 7f4a60e16640 10 monclient(hunting): authenticate will time out at 6132.932911s 2024-03-25T19:31:35.578 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.578+0000 7f4a5ad9d640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4a5c14be90 0x7f4a5c1e2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:51530/0 (socket says 172.21.15.123:51530) 2024-03-25T19:31:35.578 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.578+0000 7f4a5ad9d640 1 -- 172.21.15.123:0/2861457949 learned_addr learned my addr 172.21.15.123:0/2861457949 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:35.579 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.579+0000 7f4a52ffd640 10 client.?.objecter ms_handle_connect 0x7f4a5c14be90 2024-03-25T19:31:35.579 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.579+0000 7f4a52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:35.584 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.584+0000 7f4a52ffd640 10 client.?.objecter ms_handle_connect 0x7f4a5c0b4d60 2024-03-25T19:31:35.584 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.584+0000 7f4a52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:35.593 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.593+0000 7f4a52ffd640 10 client.?.objecter ms_handle_connect 0x7f4a5c0b1730 2024-03-25T19:31:35.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.593+0000 7f4a52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:35.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606078534 0 0) 0x7f4a5c142050 con 0x7f4a5c14be90 2024-03-25T19:31:35.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:35.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:35.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:35.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:31:35.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a3c004eb0 con 0x7f4a5c14be90 2024-03-25T19:31:35.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523347325 0 0) 0x7f4a5c140d00 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:35.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:35.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:35.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:31:35.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.595+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a5c142050 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.603+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866564153 0 0) 0x7f4a5c1dac50 con 0x7f4a5c0b1730 2024-03-25T19:31:35.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.603+0000 7f4a52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:35.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.603+0000 7f4a52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:35.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.603+0000 7f4a52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:35.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.603+0000 7f4a52ffd640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:31:35.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.603+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a5c140d00 con 0x7f4a5c0b1730 2024-03-25T19:31:35.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.615+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 148557580 0 0) 0x7f4a5c142050 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.615+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a5c1dac50 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.615+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4a44001970 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.615+0000 7f4a52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.615+0000 7f4a52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.615+0000 7f4a52ffd640 10 monclient(hunting): dump: 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:35.616 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:35.617 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:35.617 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.615+0000 7f4a52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:35.621 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.621+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 855405228 0 0) 0x7f4a3c004eb0 con 0x7f4a5c14be90 2024-03-25T19:31:35.621 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.621+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a5c142050 con 0x7f4a5c14be90 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.621+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4a54001c70 con 0x7f4a5c14be90 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.621+0000 7f4a52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.621+0000 7f4a52ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.621+0000 7f4a52ffd640 10 monclient(hunting): dump: 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:35.622 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:35.629 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1125348363 0 0) 0x7f4a5c1dac50 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 >> v1:172.21.15.151:6789/0 conn(0x7f4a5c0b1730 legacy=0x7f4a5c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 >> v1:172.21.15.123:6789/0 conn(0x7f4a5c14be90 legacy=0x7f4a5c1e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 1 monclient: found mon.c 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c1423e0 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:35.630 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a60e16640 5 monclient: authenticate success, global_id 4646 2024-03-25T19:31:35.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a60e16640 10 monclient: _renew_subs 2024-03-25T19:31:35.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a60e16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:31:35.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a60e16640 1 -- 172.21.15.123:0/2861457949 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c13b470 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a60e16640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:35.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a60e16640 10 monclient: _renew_subs 2024-03-25T19:31:35.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a60e16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:31:35.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a60e16640 1 -- 172.21.15.123:0/2861457949 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c1dba40 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.628+0000 7f4a60e16640 1 librados: init done 2024-03-25T19:31:35.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.629+0000 7f4a60e16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:35.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.630+0000 7f4a60e16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:35.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.630+0000 7f4a60e16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:31:35.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.630+0000 7f4a60e16640 1 -- 172.21.15.123:0/2861457949 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c1401f0 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.936+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a440047e0 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.936+0000 7f4a52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:35.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.936+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f4a44004af0 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.936+0000 7f4a52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:35.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.936+0000 7f4a52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:35.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.936+0000 7f4a52ffd640 10 monclient: dump: 2024-03-25T19:31:35.936 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:35.936 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:35.936 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:35.937 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:35.937 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:35.937 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:35.937 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:35.937 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:35.937 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:35.937 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:35.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.936+0000 7f4a5959a640 4 set_mon_vals no callback set 2024-03-25T19:31:35.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.937+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f4a4401aac0 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.937+0000 7f4a5a59c640 10 monclient: get_auth_request con 0x7f4a3c06df20 auth_method 0 2024-03-25T19:31:35.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.938+0000 7f4a52ffd640 10 client.4646.objecter ms_handle_connect 0x7f4a3c06df20 2024-03-25T19:31:35.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.945+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f4a44054790 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.945+0000 7f4a52ffd640 10 client.4646.objecter ms_dispatch 0x7f4a5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:35.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.945+0000 7f4a52ffd640 3 client.4646.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:35.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.945+0000 7f4a52ffd640 3 client.4646.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:35.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.945+0000 7f4a52ffd640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-25T19:31:35.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.945+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a440502a0 con 0x7f4a5c0b4d60 2024-03-25T19:31:35.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.945+0000 7f4a52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:35.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:35.945+0000 7f4a52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:36.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.203+0000 7f4a60e16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:36.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.203+0000 7f4a60e16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:36.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.203+0000 7f4a60e16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:31:36.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.203+0000 7f4a60e16640 1 -- 172.21.15.123:0/2861457949 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a5c140e90 con 0x7f4a5c0b4d60 2024-03-25T19:31:36.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.225+0000 7f4a52ffd640 1 -- 172.21.15.123:0/2861457949 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f4a440521d0 con 0x7f4a5c0b4d60 2024-03-25T19:31:36.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.225+0000 7f4a52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:36.225 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.225+0000 7f4a52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:36.225 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:31:36.225 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 10 librados: watch_flush enter 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 10 librados: watch_flush exit 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 10 client.4646.objecter successfully canceled tick 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 1 -- 172.21.15.123:0/2861457949 >> v1:172.21.15.151:6800/727684239 conn(0x7f4a3c06df20 legacy=0x7f4a3c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 10 monclient: shutdown 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 1 -- 172.21.15.123:0/2861457949 >> v1:172.21.15.123:6790/0 conn(0x7f4a5c0b4d60 legacy=0x7f4a5c1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 1 -- 172.21.15.123:0/2861457949 shutdown_connections 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 1 -- 172.21.15.123:0/2861457949 >> 172.21.15.123:0/2861457949 conn(0x7f4a5c0617a0 msgr2=0x7f4a5c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 1 -- 172.21.15.123:0/2861457949 shutdown_connections 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.227+0000 7f4a60e16640 1 -- 172.21.15.123:0/2861457949 wait complete. 2024-03-25T19:31:36.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:36.228+0000 7f4a60e16640 1 librados: shutdown 2024-03-25T19:31:36.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:36.552 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:09:26.125 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:36.557 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:09:26.130 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:36.589 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:09:26.162 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:36.607 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:09:26.140 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:36.615 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:09:26.188 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:36.619 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:09:26.153 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:36.621 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:09:26.194 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:36.652 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:09:26.185 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:36.656 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:09:26.229 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:43.047 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:43.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.163+0000 7f09cb564640 10 monclient: get_monmap_and_config 2024-03-25T19:31:43.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.163+0000 7f09cb564640 10 monclient: build_initial_monmap 2024-03-25T19:31:43.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.163+0000 7f09cb564640 10 monclient: monmap: 2024-03-25T19:31:43.163 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:43.163 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:43.164 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:31:43.164314+0000 2024-03-25T19:31:43.164 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:31:43.164314+0000 2024-03-25T19:31:43.164 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:43.164 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:43.164 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:43.164 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:31:43.164 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:31:43.164 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:43.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.164+0000 7f09cb564640 1 Processor -- start 2024-03-25T19:31:43.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.164+0000 7f09cb564640 1 -- start start 2024-03-25T19:31:43.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.164+0000 7f09cb564640 10 monclient: init 2024-03-25T19:31:43.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.164+0000 7f09cb564640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:43.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.164+0000 7f09cb564640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:43.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.165+0000 7f09cb564640 10 monclient(hunting): picked mon.noname-b con 0x7f09c40a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:43.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.165+0000 7f09cb564640 10 monclient(hunting): picked mon.noname-a con 0x7f09c413f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:43.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.165+0000 7f09cb564640 10 monclient(hunting): picked mon.noname-c con 0x7f09c41430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:43.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.165+0000 7f09cb564640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c4142520 con 0x7f09c413f710 2024-03-25T19:31:43.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.165+0000 7f09cb564640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c40b28a0 con 0x7f09c41430c0 2024-03-25T19:31:43.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.165+0000 7f09cb564640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c40b2a30 con 0x7f09c40a98f0 2024-03-25T19:31:43.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.165+0000 7f09cb564640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:43.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.165+0000 7f09cb564640 10 monclient(hunting): authenticate will time out at 6140.525129s 2024-03-25T19:31:43.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.170+0000 7f09c9300640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f09c40a98f0 0x7f09c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:38056/0 (socket says 172.21.15.123:38056) 2024-03-25T19:31:43.170 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.170+0000 7f09c9300640 1 -- 172.21.15.123:0/637258306 learned_addr learned my addr 172.21.15.123:0/637258306 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:43.177 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.177+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788925137 0 0) 0x7f09c40b2a30 con 0x7f09c40a98f0 2024-03-25T19:31:43.177 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.177+0000 7f09bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.177 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.177+0000 7f09bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.177 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.177+0000 7f09bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:43.177 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.177+0000 7f09bbfff640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:31:43.177 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.177+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b0003520 con 0x7f09c40a98f0 2024-03-25T19:31:43.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.186+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755166373 0 0) 0x7f09c4142520 con 0x7f09c413f710 2024-03-25T19:31:43.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.186+0000 7f09bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.186+0000 7f09bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.186+0000 7f09bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:43.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.186+0000 7f09bbfff640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:31:43.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.186+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c40b2a30 con 0x7f09c413f710 2024-03-25T19:31:43.190 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.190+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546550541 0 0) 0x7f09c40b28a0 con 0x7f09c41430c0 2024-03-25T19:31:43.190 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.190+0000 7f09bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.190 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.190+0000 7f09bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.190 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.190+0000 7f09bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:43.190 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.190+0000 7f09bbfff640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:31:43.190 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.190+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c4142520 con 0x7f09c41430c0 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3454884497 0 0) 0x7f09b0003520 con 0x7f09c40a98f0 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 >> v1:172.21.15.151:6789/0 conn(0x7f09c41430c0 legacy=0x7f09c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 >> v1:172.21.15.123:6789/0 conn(0x7f09c413f710 legacy=0x7f09c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 1 monclient: found mon.noname-b 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c40b44c0 con 0x7f09c40a98f0 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 10 monclient: _finish_auth 0 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f09c0001690 con 0x7f09c40a98f0 2024-03-25T19:31:43.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.195+0000 7f09bbfff640 10 monclient: dump: 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.196+0000 7f09cb564640 5 monclient: authenticate success, global_id 4649 2024-03-25T19:31:43.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.196+0000 7f09cb564640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:43.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.205+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09c00017f0 con 0x7f09c40a98f0 2024-03-25T19:31:43.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.205+0000 7f09bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:43.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.205+0000 7f09bbfff640 1 -- 172.21.15.123:0/637258306 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f09c0001c00 con 0x7f09c40a98f0 2024-03-25T19:31:43.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.205+0000 7f09bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.205+0000 7f09bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:43.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.205+0000 7f09bbfff640 10 monclient: dump: 2024-03-25T19:31:43.205 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.205+0000 7f09cb564640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:43.206 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.205+0000 7f09cb564640 4 set_mon_vals no callback set 2024-03-25T19:31:43.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.206+0000 7f09cb564640 10 monclient: shutdown 2024-03-25T19:31:43.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.206+0000 7f09cb564640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:43.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.206+0000 7f09cb564640 1 -- 172.21.15.123:0/637258306 >> v1:172.21.15.123:6790/0 conn(0x7f09c40a98f0 legacy=0x7f09c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.206+0000 7f09cb564640 1 -- 172.21.15.123:0/637258306 shutdown_connections 2024-03-25T19:31:43.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.206+0000 7f09cb564640 1 -- 172.21.15.123:0/637258306 >> 172.21.15.123:0/637258306 conn(0x7f09c40b7ea0 msgr2=0x7f09c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:43.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.206+0000 7f09cb564640 1 -- 172.21.15.123:0/637258306 shutdown_connections 2024-03-25T19:31:43.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.206+0000 7f09cb564640 1 -- 172.21.15.123:0/637258306 wait complete. 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.207+0000 7f09cb564640 10 monclient: build_initial_monmap 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.207+0000 7f09cb564640 10 monclient: monmap: 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 1 librados: starting msgr at 2024-03-25T19:31:43.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 1 librados: starting objecter 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 1 Processor -- start 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 1 -- start start 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 1 librados: setting wanted keys 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 1 librados: calling monclient init 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 10 monclient: init 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 10 monclient(hunting): picked mon.noname-a con 0x7f09c40a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 10 monclient(hunting): picked mon.noname-c con 0x7f09c413f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 10 monclient(hunting): picked mon.noname-b con 0x7f09c41430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c40abc30 con 0x7f09c40a98f0 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c40abf10 con 0x7f09c41430c0 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c4138fe0 con 0x7f09c413f710 2024-03-25T19:31:43.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:43.211 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.208+0000 7f09cb564640 10 monclient(hunting): authenticate will time out at 6140.568676s 2024-03-25T19:31:43.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.218+0000 7f09c9300640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f09c40a98f0 0x7f09c4136540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:56372/0 (socket says 172.21.15.123:56372) 2024-03-25T19:31:43.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.218+0000 7f09c9300640 1 -- 172.21.15.123:0/2448226515 learned_addr learned my addr 172.21.15.123:0/2448226515 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:43.219 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.219+0000 7f09b9ffb640 10 client.?.objecter ms_handle_connect 0x7f09c40a98f0 2024-03-25T19:31:43.219 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.220+0000 7f09b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:43.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.221+0000 7f09b9ffb640 10 client.?.objecter ms_handle_connect 0x7f09c413f710 2024-03-25T19:31:43.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.221+0000 7f09b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:43.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.228+0000 7f09b9ffb640 10 client.?.objecter ms_handle_connect 0x7f09c41430c0 2024-03-25T19:31:43.228 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.228+0000 7f09b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:43.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.239+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193120742 0 0) 0x7f09c4138fe0 con 0x7f09c413f710 2024-03-25T19:31:43.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.239+0000 7f09b9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.239+0000 7f09b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.239+0000 7f09b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:43.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.239+0000 7f09b9ffb640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:31:43.239 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.239+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a8004eb0 con 0x7f09c413f710 2024-03-25T19:31:43.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.249+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434367951 0 0) 0x7f09c40abc30 con 0x7f09c40a98f0 2024-03-25T19:31:43.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.249+0000 7f09b9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.249+0000 7f09b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.249+0000 7f09b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:43.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.249+0000 7f09b9ffb640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:31:43.249 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.249+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c4138fe0 con 0x7f09c40a98f0 2024-03-25T19:31:43.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.252+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803936292 0 0) 0x7f09c40abf10 con 0x7f09c41430c0 2024-03-25T19:31:43.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.252+0000 7f09b9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.252+0000 7f09b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.252+0000 7f09b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:43.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.252+0000 7f09b9ffb640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:31:43.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.252+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c40abc30 con 0x7f09c41430c0 2024-03-25T19:31:43.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.268+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1519117465 0 0) 0x7f09c40abc30 con 0x7f09c41430c0 2024-03-25T19:31:43.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.268+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09c40abf10 con 0x7f09c41430c0 2024-03-25T19:31:43.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.268+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f09bc001bd0 con 0x7f09c41430c0 2024-03-25T19:31:43.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.268+0000 7f09b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.268 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.268+0000 7f09b9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.268+0000 7f09b9ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:43.269 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.268+0000 7f09b9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:43.279 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.279+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3377523330 0 0) 0x7f09c4138fe0 con 0x7f09c40a98f0 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.279+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09c40abc30 con 0x7f09c40a98f0 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.279+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f09c0001b00 con 0x7f09c40a98f0 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.279+0000 7f09b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.279+0000 7f09b9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.279+0000 7f09b9ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:43.280 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.280+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3001192285 0 0) 0x7f09a8004eb0 con 0x7f09c413f710 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.280+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09a8005ed0 con 0x7f09c413f710 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.280+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f09b4001970 con 0x7f09c413f710 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.280+0000 7f09b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.280+0000 7f09b9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.280+0000 7f09b9ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:43.281 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:43.282 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1994886335 0 0) 0x7f09c40abf10 con 0x7f09c41430c0 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 >> v1:172.21.15.123:6790/0 conn(0x7f09c413f710 legacy=0x7f09c41387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 >> v1:172.21.15.123:6789/0 conn(0x7f09c40a98f0 legacy=0x7f09c4136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 1 monclient: found mon.b 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c4137bf0 con 0x7f09c41430c0 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:43.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:43.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09cb564640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:31:43.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.286+0000 7f09cb564640 10 monclient: _renew_subs 2024-03-25T19:31:43.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.287+0000 7f09cb564640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:43.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.287+0000 7f09cb564640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c4137e40 con 0x7f09c41430c0 2024-03-25T19:31:43.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.287+0000 7f09cb564640 10 client.4633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:43.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.287+0000 7f09cb564640 10 monclient: _renew_subs 2024-03-25T19:31:43.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.287+0000 7f09cb564640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:43.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.287+0000 7f09cb564640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c41e3690 con 0x7f09c41430c0 2024-03-25T19:31:43.289 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.287+0000 7f09cb564640 1 librados: init done 2024-03-25T19:31:43.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.288+0000 7f09cb564640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:43.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.288+0000 7f09cb564640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:43.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.288+0000 7f09cb564640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:43.290 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.288+0000 7f09cb564640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c40627b0 con 0x7f09c41430c0 2024-03-25T19:31:43.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.574+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09bc001fe0 con 0x7f09c41430c0 2024-03-25T19:31:43.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.574+0000 7f09b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:43.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.574+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f09bc003b30 con 0x7f09c41430c0 2024-03-25T19:31:43.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.574+0000 7f09b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.574+0000 7f09b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:43.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.574+0000 7f09b9ffb640 10 monclient: dump: 2024-03-25T19:31:43.574 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:43.574 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:43.574 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.575 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:43.575 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.575 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:43.575 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:43.575 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:43.575 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:43.575 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:43.575 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.574+0000 7f09bb7fe640 4 set_mon_vals no callback set 2024-03-25T19:31:43.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.575+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f09bc019b00 con 0x7f09c41430c0 2024-03-25T19:31:43.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.575+0000 7f09c9300640 10 monclient: get_auth_request con 0x7f09a806e3f0 auth_method 0 2024-03-25T19:31:43.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.576+0000 7f09b9ffb640 10 client.4633.objecter ms_handle_connect 0x7f09a806e3f0 2024-03-25T19:31:43.579 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.580+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f09bc053680 con 0x7f09c41430c0 2024-03-25T19:31:43.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.580+0000 7f09b9ffb640 10 client.4633.objecter ms_dispatch 0x7f09c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:43.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.580+0000 7f09b9ffb640 3 client.4633.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:43.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.580+0000 7f09b9ffb640 3 client.4633.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:43.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.580+0000 7f09b9ffb640 20 client.4633.objecter dump_active .. 0 homeless 2024-03-25T19:31:43.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.580+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09bc04f190 con 0x7f09c41430c0 2024-03-25T19:31:43.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.580+0000 7f09b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:43.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.580+0000 7f09b9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.826+0000 7f09cb564640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.826+0000 7f09cb564640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.826+0000 7f09cb564640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.826+0000 7f09cb564640 1 -- 172.21.15.123:0/2448226515 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09c413e250 con 0x7f09c41430c0 2024-03-25T19:31:43.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.858+0000 7f09b9ffb640 1 -- 172.21.15.123:0/2448226515 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f09bc0510c0 con 0x7f09c41430c0 2024-03-25T19:31:43.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.858+0000 7f09b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:43.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.858+0000 7f09b9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:43.858 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:31:43.858 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.860+0000 7f09cb564640 10 librados: watch_flush enter 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.860+0000 7f09cb564640 10 librados: watch_flush exit 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.860+0000 7f09cb564640 20 client.4633.objecter shutdown clearing up homeless session... 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.860+0000 7f09cb564640 10 client.4633.objecter successfully canceled tick 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.860+0000 7f09cb564640 1 -- 172.21.15.123:0/2448226515 >> v1:172.21.15.151:6800/727684239 conn(0x7f09a806e3f0 legacy=0x7f09a80708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.860+0000 7f09cb564640 10 monclient: shutdown 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.860+0000 7f09cb564640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.860+0000 7f09cb564640 1 -- 172.21.15.123:0/2448226515 >> v1:172.21.15.151:6789/0 conn(0x7f09c41430c0 legacy=0x7f09c40ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.860+0000 7f09cb564640 1 -- 172.21.15.123:0/2448226515 shutdown_connections 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.860+0000 7f09cb564640 1 -- 172.21.15.123:0/2448226515 >> 172.21.15.123:0/2448226515 conn(0x7f09c40b7ea0 msgr2=0x7f09c413c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.861+0000 7f09cb564640 1 -- 172.21.15.123:0/2448226515 shutdown_connections 2024-03-25T19:31:43.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.861+0000 7f09cb564640 1 -- 172.21.15.123:0/2448226515 wait complete. 2024-03-25T19:31:43.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:43.861+0000 7f09cb564640 1 librados: shutdown 2024-03-25T19:31:43.872 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:50.678 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:50.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.789+0000 7f1b5f596640 10 monclient: get_monmap_and_config 2024-03-25T19:31:50.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.789+0000 7f1b5f596640 10 monclient: build_initial_monmap 2024-03-25T19:31:50.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.790+0000 7f1b5f596640 10 monclient: monmap: 2024-03-25T19:31:50.790 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:50.790 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:50.790 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:31:50.790895+0000 2024-03-25T19:31:50.790 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:31:50.790895+0000 2024-03-25T19:31:50.790 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:50.790 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:50.790 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:50.791 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:31:50.791 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:31:50.791 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:50.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.790+0000 7f1b5f596640 1 Processor -- start 2024-03-25T19:31:50.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.790+0000 7f1b5f596640 1 -- start start 2024-03-25T19:31:50.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.790+0000 7f1b5f596640 10 monclient: init 2024-03-25T19:31:50.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5f596640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:50.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5f596640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:50.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5f596640 10 monclient(hunting): picked mon.noname-a con 0x7f1b5813fc60 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:50.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5f596640 10 monclient(hunting): picked mon.noname-b con 0x7f1b58070570 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:50.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5f596640 10 monclient(hunting): picked mon.noname-c con 0x7f1b580727a0 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:50.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5f596640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5813c9f0 con 0x7f1b5813fc60 2024-03-25T19:31:50.792 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5f596640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5806fac0 con 0x7f1b580727a0 2024-03-25T19:31:50.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5f596640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b580715d0 con 0x7f1b58070570 2024-03-25T19:31:50.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5f596640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:50.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5f596640 10 monclient(hunting): authenticate will time out at 6148.151374s 2024-03-25T19:31:50.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5cb31640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f1b58070570 0x7f1b58072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:57468/0 (socket says 172.21.15.123:57468) 2024-03-25T19:31:50.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.791+0000 7f1b5cb31640 1 -- 172.21.15.123:0/2472226066 learned_addr learned my addr 172.21.15.123:0/2472226066 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:50.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.861+0000 7f1b4ffff640 1 -- 172.21.15.123:0/2472226066 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848587021 0 0) 0x7f1b5813c9f0 con 0x7f1b5813fc60 2024-03-25T19:31:50.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.861+0000 7f1b4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:50.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.861+0000 7f1b4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:50.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.861+0000 7f1b4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:50.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.861+0000 7f1b4ffff640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:31:50.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.861+0000 7f1b4ffff640 1 -- 172.21.15.123:0/2472226066 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b44003520 con 0x7f1b5813fc60 2024-03-25T19:31:50.884 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 1 -- 172.21.15.123:0/2472226066 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2119887706 0 0) 0x7f1b44003520 con 0x7f1b5813fc60 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 1 -- 172.21.15.123:0/2472226066 >> v1:172.21.15.123:6790/0 conn(0x7f1b58070570 legacy=0x7f1b58072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 1 -- 172.21.15.123:0/2472226066 >> v1:172.21.15.151:6789/0 conn(0x7f1b580727a0 legacy=0x7f1b5814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 1 monclient: found mon.noname-a 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 1 -- 172.21.15.123:0/2472226066 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b580718b0 con 0x7f1b5813fc60 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 10 monclient: _finish_auth 0 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 1 -- 172.21.15.123:0/2472226066 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1b54001690 con 0x7f1b5813fc60 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:50.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.884+0000 7f1b4ffff640 10 monclient: dump: 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.885+0000 7f1b5f596640 5 monclient: authenticate success, global_id 4626 2024-03-25T19:31:50.886 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.885+0000 7f1b5f596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b4ffff640 1 -- 172.21.15.123:0/2472226066 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b540017f0 con 0x7f1b5813fc60 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b4ffff640 1 -- 172.21.15.123:0/2472226066 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1b54001c00 con 0x7f1b5813fc60 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b5f596640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b5f596640 4 set_mon_vals no callback set 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b4ffff640 10 monclient: dump: 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:50.898 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:50.899 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:50.899 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:50.899 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:50.899 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:50.899 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:50.899 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:50.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b5f596640 10 monclient: shutdown 2024-03-25T19:31:50.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b5f596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:50.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b5f596640 1 -- 172.21.15.123:0/2472226066 >> v1:172.21.15.123:6789/0 conn(0x7f1b5813fc60 legacy=0x7f1b5806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:50.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b5f596640 1 -- 172.21.15.123:0/2472226066 shutdown_connections 2024-03-25T19:31:50.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.898+0000 7f1b5f596640 1 -- 172.21.15.123:0/2472226066 >> 172.21.15.123:0/2472226066 conn(0x7f1b580b4a80 msgr2=0x7f1b580b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:50.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.899+0000 7f1b5f596640 1 -- 172.21.15.123:0/2472226066 shutdown_connections 2024-03-25T19:31:50.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.899+0000 7f1b5f596640 1 -- 172.21.15.123:0/2472226066 wait complete. 2024-03-25T19:31:50.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.899+0000 7f1b5f596640 10 monclient: build_initial_monmap 2024-03-25T19:31:50.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.899+0000 7f1b5f596640 10 monclient: monmap: 2024-03-25T19:31:50.901 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.900+0000 7f1b5f596640 1 librados: starting msgr at 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.900+0000 7f1b5f596640 1 librados: starting objecter 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.900+0000 7f1b5f596640 1 Processor -- start 2024-03-25T19:31:50.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.900+0000 7f1b5f596640 1 -- start start 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.900+0000 7f1b5f596640 1 librados: setting wanted keys 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.900+0000 7f1b5f596640 1 librados: calling monclient init 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.900+0000 7f1b5f596640 10 monclient: init 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.901+0000 7f1b5f596640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.901+0000 7f1b5f596640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.901+0000 7f1b5f596640 10 monclient(hunting): picked mon.noname-a con 0x7f1b58070570 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.901+0000 7f1b5f596640 10 monclient(hunting): picked mon.noname-b con 0x7f1b580727a0 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.901+0000 7f1b5f596640 10 monclient(hunting): picked mon.noname-c con 0x7f1b5813fc60 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.901+0000 7f1b5f596640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b580ae0e0 con 0x7f1b58070570 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.901+0000 7f1b5f596640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b58138b40 con 0x7f1b580727a0 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.901+0000 7f1b5f596640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b58138cd0 con 0x7f1b5813fc60 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.901+0000 7f1b5f596640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:50.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.901+0000 7f1b5f596640 10 monclient(hunting): authenticate will time out at 6148.261201s 2024-03-25T19:31:50.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.906+0000 7f1b5db33640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f1b5813fc60 0x7f1b581d7920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:57512/0 (socket says 172.21.15.123:57512) 2024-03-25T19:31:50.906 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.906+0000 7f1b5db33640 1 -- 172.21.15.123:0/1203481987 learned_addr learned my addr 172.21.15.123:0/1203481987 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:50.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.909+0000 7f1b4dffb640 10 client.?.objecter ms_handle_connect 0x7f1b5813fc60 2024-03-25T19:31:50.908 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.909+0000 7f1b4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:50.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.916+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879292395 0 0) 0x7f1b58138cd0 con 0x7f1b5813fc60 2024-03-25T19:31:50.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.916+0000 7f1b4dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:50.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.916+0000 7f1b4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:50.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.916+0000 7f1b4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:50.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.916+0000 7f1b4dffb640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:31:50.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.916+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b3c004eb0 con 0x7f1b5813fc60 2024-03-25T19:31:50.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.917+0000 7f1b4dffb640 10 client.?.objecter ms_handle_connect 0x7f1b58070570 2024-03-25T19:31:50.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.917+0000 7f1b4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:50.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.921+0000 7f1b4dffb640 10 client.?.objecter ms_handle_connect 0x7f1b580727a0 2024-03-25T19:31:50.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.921+0000 7f1b4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:50.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.928+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 244605700 0 0) 0x7f1b3c004eb0 con 0x7f1b5813fc60 2024-03-25T19:31:50.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.928+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b58138cd0 con 0x7f1b5813fc60 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.928+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1b50001c70 con 0x7f1b5813fc60 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.928+0000 7f1b4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.928+0000 7f1b4dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.928+0000 7f1b4dffb640 10 monclient(hunting): dump: 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:50.928 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:50.929 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.928+0000 7f1b4dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:50.930 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.931+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507867624 0 0) 0x7f1b58138b40 con 0x7f1b580727a0 2024-03-25T19:31:50.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.931+0000 7f1b4dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:50.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.931+0000 7f1b4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:50.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.931+0000 7f1b4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:50.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.931+0000 7f1b4dffb640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:31:50.931 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.931+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b3c004eb0 con 0x7f1b580727a0 2024-03-25T19:31:50.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.939+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114455734 0 0) 0x7f1b580ae0e0 con 0x7f1b58070570 2024-03-25T19:31:50.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.939+0000 7f1b4dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:50.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.939+0000 7f1b4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:50.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.939+0000 7f1b4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:50.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.939+0000 7f1b4dffb640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:31:50.939 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.939+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b58138b40 con 0x7f1b58070570 2024-03-25T19:31:50.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.940+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2174155991 0 0) 0x7f1b3c004eb0 con 0x7f1b580727a0 2024-03-25T19:31:50.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.940+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b580ae0e0 con 0x7f1b580727a0 2024-03-25T19:31:50.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.940+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1b48001860 con 0x7f1b580727a0 2024-03-25T19:31:50.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.940+0000 7f1b4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:50.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.940+0000 7f1b4dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:50.940 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.940+0000 7f1b4dffb640 10 monclient(hunting): dump: 2024-03-25T19:31:50.941 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:50.941 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:50.941 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:50.941 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:50.941 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:50.941 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:50.941 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:50.941 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:50.941 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:50.941 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:50.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.941+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3397074875 0 0) 0x7f1b58138cd0 con 0x7f1b5813fc60 2024-03-25T19:31:50.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.941+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 >> v1:172.21.15.151:6789/0 conn(0x7f1b580727a0 legacy=0x7f1b58132fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:50.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.941+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 >> v1:172.21.15.123:6789/0 conn(0x7f1b58070570 legacy=0x7f1b580ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:50.942 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.941+0000 7f1b4dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b4dffb640 1 monclient: found mon.c 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b4dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b581336e0 con 0x7f1b5813fc60 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b4dffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b4dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b5f596640 5 monclient: authenticate success, global_id 4658 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b5f596640 10 monclient: _renew_subs 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b5f596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b5f596640 1 -- 172.21.15.123:0/1203481987 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b58133930 con 0x7f1b5813fc60 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b5f596640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:50.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b5f596640 10 monclient: _renew_subs 2024-03-25T19:31:50.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b5f596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:31:50.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b5f596640 1 -- 172.21.15.123:0/1203481987 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b58139a30 con 0x7f1b5813fc60 2024-03-25T19:31:50.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.942+0000 7f1b5f596640 1 librados: init done 2024-03-25T19:31:50.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.943+0000 7f1b5f596640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:50.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.943+0000 7f1b5f596640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:50.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.943+0000 7f1b5f596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:31:50.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:50.943+0000 7f1b5f596640 1 -- 172.21.15.123:0/1203481987 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b580627b0 con 0x7f1b5813fc60 2024-03-25T19:31:51.241 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.240+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b50003990 con 0x7f1b5813fc60 2024-03-25T19:31:51.241 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.240+0000 7f1b4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:51.241 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.240+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f1b50003da0 con 0x7f1b5813fc60 2024-03-25T19:31:51.241 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.240+0000 7f1b4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:51.241 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.240+0000 7f1b4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:51.241 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.240+0000 7f1b4dffb640 10 monclient: dump: 2024-03-25T19:31:51.241 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:51.241 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:51.241 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:51.241 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:51.242 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:51.242 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:51.242 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:51.242 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:51.242 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:51.242 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:51.242 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.240+0000 7f1b4f7fe640 4 set_mon_vals no callback set 2024-03-25T19:31:51.242 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.241+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f1b50003fc0 con 0x7f1b5813fc60 2024-03-25T19:31:51.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.242+0000 7f1b5d332640 10 monclient: get_auth_request con 0x7f1b3c06e020 auth_method 0 2024-03-25T19:31:51.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.242+0000 7f1b4dffb640 10 client.4658.objecter ms_handle_connect 0x7f1b3c06e020 2024-03-25T19:31:51.250 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.250+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f1b50054ad0 con 0x7f1b5813fc60 2024-03-25T19:31:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.250+0000 7f1b4dffb640 10 client.4658.objecter ms_dispatch 0x7f1b58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.250+0000 7f1b4dffb640 3 client.4658.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.250+0000 7f1b4dffb640 3 client.4658.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.250+0000 7f1b4dffb640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-25T19:31:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.250+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b500505e0 con 0x7f1b5813fc60 2024-03-25T19:31:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.250+0000 7f1b4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:51.251 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.250+0000 7f1b4dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:51.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.499+0000 7f1b5f596640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:51.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.499+0000 7f1b5f596640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:51.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.499+0000 7f1b5f596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:31:51.499 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.499+0000 7f1b5f596640 1 -- 172.21.15.123:0/1203481987 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b580af160 con 0x7f1b5813fc60 2024-03-25T19:31:51.517 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.517+0000 7f1b4dffb640 1 -- 172.21.15.123:0/1203481987 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f1b50052510 con 0x7f1b5813fc60 2024-03-25T19:31:51.517 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.517+0000 7f1b4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:51.517 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.517+0000 7f1b4dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:51.518 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:31:51.518 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:51.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 10 librados: watch_flush enter 2024-03-25T19:31:51.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 10 librados: watch_flush exit 2024-03-25T19:31:51.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-25T19:31:51.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 10 client.4658.objecter successfully canceled tick 2024-03-25T19:31:51.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 1 -- 172.21.15.123:0/1203481987 >> v1:172.21.15.151:6800/727684239 conn(0x7f1b3c06e020 legacy=0x7f1b3c0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 10 monclient: shutdown 2024-03-25T19:31:51.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:51.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 1 -- 172.21.15.123:0/1203481987 >> v1:172.21.15.123:6790/0 conn(0x7f1b5813fc60 legacy=0x7f1b581d7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 1 -- 172.21.15.123:0/1203481987 shutdown_connections 2024-03-25T19:31:51.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 1 -- 172.21.15.123:0/1203481987 >> 172.21.15.123:0/1203481987 conn(0x7f1b580b4a80 msgr2=0x7f1b580b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:51.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 1 -- 172.21.15.123:0/1203481987 shutdown_connections 2024-03-25T19:31:51.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 1 -- 172.21.15.123:0/1203481987 wait complete. 2024-03-25T19:31:51.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:51.520+0000 7f1b5f596640 1 librados: shutdown 2024-03-25T19:31:51.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:56.506 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:09:46.078 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:56.554 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:09:46.126 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:56.585 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:09:46.157 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:56.608 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:09:46.180 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:56.609 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:09:46.143 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:56.615 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:09:46.187 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:56.615 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:09:46.149 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:56.642 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:09:46.176 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:56.653 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:09:46.225 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:58.339 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:58.453 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.453+0000 7fd93d5a4640 10 monclient: get_monmap_and_config 2024-03-25T19:31:58.453 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.453+0000 7fd93d5a4640 10 monclient: build_initial_monmap 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.453+0000 7fd93d5a4640 10 monclient: monmap: 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:31:58.454591+0000 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:31:58.454591+0000 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:31:58.454 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:58.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.454+0000 7fd93d5a4640 1 Processor -- start 2024-03-25T19:31:58.455 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.454+0000 7fd93d5a4640 1 -- start start 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.454+0000 7fd93d5a4640 10 monclient: init 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.455+0000 7fd93d5a4640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.455+0000 7fd93d5a4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.455+0000 7fd93d5a4640 10 monclient(hunting): picked mon.noname-a con 0x7fd9380a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.455+0000 7fd93d5a4640 10 monclient(hunting): picked mon.noname-c con 0x7fd93813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.455+0000 7fd93d5a4640 10 monclient(hunting): picked mon.noname-b con 0x7fd9381430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.455+0000 7fd93d5a4640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd938142520 con 0x7fd9380a98f0 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.455+0000 7fd93d5a4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380b28a0 con 0x7fd93813f710 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.455+0000 7fd93d5a4640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380b2a30 con 0x7fd9381430c0 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.455+0000 7fd93d5a4640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:58.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.455+0000 7fd93d5a4640 10 monclient(hunting): authenticate will time out at 6155.815345s 2024-03-25T19:31:58.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.457+0000 7fd936ffd640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fd9380a98f0 0x7fd9380a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46746/0 (socket says 172.21.15.123:46746) 2024-03-25T19:31:58.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.457+0000 7fd936ffd640 1 -- 172.21.15.123:0/3681168391 learned_addr learned my addr 172.21.15.123:0/3681168391 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:58.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.463+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045105730 0 0) 0x7fd938142520 con 0x7fd9380a98f0 2024-03-25T19:31:58.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.463+0000 7fd935ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.463+0000 7fd935ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.463+0000 7fd935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.463+0000 7fd935ffb640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:31:58.464 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.464+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd920003520 con 0x7fd9380a98f0 2024-03-25T19:31:58.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748621229 0 0) 0x7fd9380b2a30 con 0x7fd9381430c0 2024-03-25T19:31:58.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.472 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.473 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.473 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:31:58.473 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd938142520 con 0x7fd9381430c0 2024-03-25T19:31:58.473 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426633889 0 0) 0x7fd9380b28a0 con 0x7fd93813f710 2024-03-25T19:31:58.473 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.473 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.473 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.474 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:31:58.474 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.473+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9380b2a30 con 0x7fd93813f710 2024-03-25T19:31:58.477 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3090102297 0 0) 0x7fd920003520 con 0x7fd9380a98f0 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 >> v1:172.21.15.123:6790/0 conn(0x7fd9381430c0 legacy=0x7fd93814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 >> v1:172.21.15.151:6789/0 conn(0x7fd93813f710 legacy=0x7fd9381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 1 monclient: found mon.noname-a 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380b44c0 con 0x7fd9380a98f0 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd935ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd93d5a4640 5 monclient: authenticate success, global_id 4632 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.477+0000 7fd93d5a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:58.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.478+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd928001650 con 0x7fd9380a98f0 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.478+0000 7fd935ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.478+0000 7fd935ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.478+0000 7fd935ffb640 10 monclient: dump: 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:58.479 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.478+0000 7fd93d5a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:58.482 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9280018a0 con 0x7fd9380a98f0 2024-03-25T19:31:58.482 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd935ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd935ffb640 1 -- 172.21.15.123:0/3681168391 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd928001c30 con 0x7fd9380a98f0 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd935ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd935ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd935ffb640 10 monclient: dump: 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:58.483 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:58.484 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:58.484 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:58.484 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:58.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd93d5a4640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:58.484 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd93d5a4640 4 set_mon_vals no callback set 2024-03-25T19:31:58.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd93d5a4640 10 monclient: shutdown 2024-03-25T19:31:58.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd93d5a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd93d5a4640 1 -- 172.21.15.123:0/3681168391 >> v1:172.21.15.123:6789/0 conn(0x7fd9380a98f0 legacy=0x7fd9380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd93d5a4640 1 -- 172.21.15.123:0/3681168391 shutdown_connections 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.483+0000 7fd93d5a4640 1 -- 172.21.15.123:0/3681168391 >> 172.21.15.123:0/3681168391 conn(0x7fd9380b7ea0 msgr2=0x7fd9380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.484+0000 7fd93d5a4640 1 -- 172.21.15.123:0/3681168391 shutdown_connections 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.484+0000 7fd93d5a4640 1 -- 172.21.15.123:0/3681168391 wait complete. 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.484+0000 7fd93d5a4640 10 monclient: build_initial_monmap 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.484+0000 7fd93d5a4640 10 monclient: monmap: 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:58.487 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.484+0000 7fd93d5a4640 1 librados: starting msgr at 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.484+0000 7fd93d5a4640 1 librados: starting objecter 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.485+0000 7fd93d5a4640 1 Processor -- start 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.485+0000 7fd93d5a4640 1 -- start start 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.485+0000 7fd93d5a4640 1 librados: setting wanted keys 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.485+0000 7fd93d5a4640 1 librados: calling monclient init 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.485+0000 7fd93d5a4640 10 monclient: init 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.485+0000 7fd93d5a4640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.486+0000 7fd93d5a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.486+0000 7fd93d5a4640 10 monclient(hunting): picked mon.noname-c con 0x7fd9380a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.486+0000 7fd93d5a4640 10 monclient(hunting): picked mon.noname-b con 0x7fd93813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:31:58.488 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.486+0000 7fd93d5a4640 10 monclient(hunting): picked mon.noname-a con 0x7fd9381430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:31:58.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.486+0000 7fd93d5a4640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380b17e0 con 0x7fd9381430c0 2024-03-25T19:31:58.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.486+0000 7fd93d5a4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9381392b0 con 0x7fd93813f710 2024-03-25T19:31:58.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.486+0000 7fd93d5a4640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380abca0 con 0x7fd9380a98f0 2024-03-25T19:31:58.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.486+0000 7fd93d5a4640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:58.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.486+0000 7fd93d5a4640 10 monclient(hunting): authenticate will time out at 6155.846298s 2024-03-25T19:31:58.495 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.496+0000 7fd9377fe640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fd9381430c0 0x7fd9380ac610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:46762/0 (socket says 172.21.15.123:46762) 2024-03-25T19:31:58.495 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.496+0000 7fd9377fe640 1 -- 172.21.15.123:0/1865972359 learned_addr learned my addr 172.21.15.123:0/1865972359 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:31:58.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.496+0000 7fd91f7fe640 10 client.?.objecter ms_handle_connect 0x7fd9381430c0 2024-03-25T19:31:58.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.496+0000 7fd91f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:58.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.507+0000 7fd91f7fe640 10 client.?.objecter ms_handle_connect 0x7fd9380a98f0 2024-03-25T19:31:58.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.507+0000 7fd91f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:58.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.510+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012516807 0 0) 0x7fd938142520 con 0x7fd9381430c0 2024-03-25T19:31:58.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.510+0000 7fd91f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.510+0000 7fd91f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.510+0000 7fd91f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.510+0000 7fd91f7fe640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:31:58.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.510+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd924004eb0 con 0x7fd9381430c0 2024-03-25T19:31:58.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.511+0000 7fd91f7fe640 10 client.?.objecter ms_handle_connect 0x7fd93813f710 2024-03-25T19:31:58.511 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.511+0000 7fd91f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:58.512 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.513+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268651022 0 0) 0x7fd9380abca0 con 0x7fd9380a98f0 2024-03-25T19:31:58.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.513+0000 7fd91f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.513+0000 7fd91f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.513+0000 7fd91f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.513+0000 7fd91f7fe640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:31:58.513 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.513+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd938142520 con 0x7fd9380a98f0 2024-03-25T19:31:58.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.523+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 995701612 0 0) 0x7fd924004eb0 con 0x7fd9381430c0 2024-03-25T19:31:58.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.523+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9380abca0 con 0x7fd9381430c0 2024-03-25T19:31:58.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.523+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd930001a60 con 0x7fd9381430c0 2024-03-25T19:31:58.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.523+0000 7fd91f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.523+0000 7fd91f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:58.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.523+0000 7fd91f7fe640 10 monclient(hunting): dump: 2024-03-25T19:31:58.523 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:58.523 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:58.524 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:58.524 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:58.524 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.524 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:58.524 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:58.524 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:58.524 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:58.524 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:58.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.523+0000 7fd91f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:58.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.526+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012336963 0 0) 0x7fd9381392b0 con 0x7fd93813f710 2024-03-25T19:31:58.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.526+0000 7fd91f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.526+0000 7fd91f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.526+0000 7fd91f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.526+0000 7fd91f7fe640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:31:58.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.526+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd924004eb0 con 0x7fd93813f710 2024-03-25T19:31:58.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.535+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 590843130 0 0) 0x7fd938142520 con 0x7fd9380a98f0 2024-03-25T19:31:58.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.535+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9381392b0 con 0x7fd9380a98f0 2024-03-25T19:31:58.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.535+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd928001bb0 con 0x7fd9380a98f0 2024-03-25T19:31:58.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.535+0000 7fd91f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.535+0000 7fd91f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:58.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.535+0000 7fd91f7fe640 10 monclient(hunting): dump: 2024-03-25T19:31:58.536 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:58.536 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:58.536 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:58.536 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:58.536 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.536 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:58.536 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:58.536 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:58.536 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:58.536 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:58.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3982434056 0 0) 0x7fd9380abca0 con 0x7fd9381430c0 2024-03-25T19:31:58.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 >> v1:172.21.15.123:6790/0 conn(0x7fd9380a98f0 legacy=0x7fd938138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 >> v1:172.21.15.151:6789/0 conn(0x7fd93813f710 legacy=0x7fd9381366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:31:58.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 1 monclient: found mon.a 2024-03-25T19:31:58.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380abf10 con 0x7fd9381430c0 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd91f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd93d5a4640 5 monclient: authenticate success, global_id 4635 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd93d5a4640 10 monclient: _renew_subs 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd93d5a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd93d5a4640 1 -- 172.21.15.123:0/1865972359 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9380ac1f0 con 0x7fd9381430c0 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd93d5a4640 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd93d5a4640 10 monclient: _renew_subs 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd93d5a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd93d5a4640 1 -- 172.21.15.123:0/1865972359 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9381e34b0 con 0x7fd9381430c0 2024-03-25T19:31:58.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.537+0000 7fd93d5a4640 1 librados: init done 2024-03-25T19:31:58.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.538+0000 7fd93d5a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:58.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.538+0000 7fd93d5a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:58.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.538+0000 7fd93d5a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:31:58.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.538+0000 7fd93d5a4640 1 -- 172.21.15.123:0/1865972359 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9380627b0 con 0x7fd9381430c0 2024-03-25T19:31:58.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.828+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd930001f50 con 0x7fd9381430c0 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.828+0000 7fd91f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.828+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fd930003b10 con 0x7fd9381430c0 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.828+0000 7fd91f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.828+0000 7fd91f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.828+0000 7fd91f7fe640 10 monclient: dump: 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:31:58.829 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:31:58.830 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:31:58.830 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:31:58.830 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:31:58.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.828+0000 7fd9357fa640 4 set_mon_vals no callback set 2024-03-25T19:31:58.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.829+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fd930003d30 con 0x7fd9381430c0 2024-03-25T19:31:58.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.829+0000 7fd936ffd640 10 monclient: get_auth_request con 0x7fd92406dfb0 auth_method 0 2024-03-25T19:31:58.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.830+0000 7fd91f7fe640 10 client.4635.objecter ms_handle_connect 0x7fd92406dfb0 2024-03-25T19:31:58.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.836+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fd930054840 con 0x7fd9381430c0 2024-03-25T19:31:58.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.837+0000 7fd91f7fe640 10 client.4635.objecter ms_dispatch 0x7fd938000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:58.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.837+0000 7fd91f7fe640 3 client.4635.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:58.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.837+0000 7fd91f7fe640 3 client.4635.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:58.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.837+0000 7fd91f7fe640 20 client.4635.objecter dump_active .. 0 homeless 2024-03-25T19:31:58.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.837+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd930050350 con 0x7fd9381430c0 2024-03-25T19:31:58.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.837+0000 7fd91f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:58.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:58.837+0000 7fd91f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:59.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.061+0000 7fd93d5a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:59.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.061+0000 7fd93d5a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:59.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.061+0000 7fd93d5a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:31:59.061 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.061+0000 7fd93d5a4640 1 -- 172.21.15.123:0/1865972359 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9380b2300 con 0x7fd9381430c0 2024-03-25T19:31:59.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.144+0000 7fd91f7fe640 1 -- 172.21.15.123:0/1865972359 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fd930052280 con 0x7fd9381430c0 2024-03-25T19:31:59.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.144+0000 7fd91f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:59.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.144+0000 7fd91f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:59.144 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:31:59.144 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.146+0000 7fd93d5a4640 10 librados: watch_flush enter 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.146+0000 7fd93d5a4640 10 librados: watch_flush exit 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.146+0000 7fd93d5a4640 20 client.4635.objecter shutdown clearing up homeless session... 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.146+0000 7fd93d5a4640 10 client.4635.objecter successfully canceled tick 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.147+0000 7fd93d5a4640 1 -- 172.21.15.123:0/1865972359 >> v1:172.21.15.151:6800/727684239 conn(0x7fd92406dfb0 legacy=0x7fd924070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.147+0000 7fd93d5a4640 10 monclient: shutdown 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.147+0000 7fd93d5a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.147+0000 7fd93d5a4640 1 -- 172.21.15.123:0/1865972359 >> v1:172.21.15.123:6789/0 conn(0x7fd9381430c0 legacy=0x7fd9380ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.147+0000 7fd93d5a4640 1 -- 172.21.15.123:0/1865972359 shutdown_connections 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.147+0000 7fd93d5a4640 1 -- 172.21.15.123:0/1865972359 >> 172.21.15.123:0/1865972359 conn(0x7fd9380b7ea0 msgr2=0x7fd938074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:59.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.147+0000 7fd93d5a4640 1 -- 172.21.15.123:0/1865972359 shutdown_connections 2024-03-25T19:31:59.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.147+0000 7fd93d5a4640 1 -- 172.21.15.123:0/1865972359 wait complete. 2024-03-25T19:31:59.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:31:59.147+0000 7fd93d5a4640 1 librados: shutdown 2024-03-25T19:31:59.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:05.964 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:06.079 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.080+0000 7f57d8cc8640 10 monclient: get_monmap_and_config 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.080+0000 7f57d8cc8640 10 monclient: build_initial_monmap 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.080+0000 7f57d8cc8640 10 monclient: monmap: 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:32:06.081066+0000 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:32:06.081066+0000 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:32:06.080 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:06.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.080+0000 7f57d8cc8640 1 Processor -- start 2024-03-25T19:32:06.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.080+0000 7f57d8cc8640 1 -- start start 2024-03-25T19:32:06.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.080+0000 7f57d8cc8640 10 monclient: init 2024-03-25T19:32:06.081 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.081+0000 7f57d8cc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:06.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.081+0000 7f57d8cc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:06.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.081+0000 7f57d8cc8640 10 monclient(hunting): picked mon.noname-b con 0x7f57d40b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:06.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.081+0000 7f57d8cc8640 10 monclient(hunting): picked mon.noname-a con 0x7f57d40b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:06.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.081+0000 7f57d8cc8640 10 monclient(hunting): picked mon.noname-c con 0x7f57d414be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:06.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.081+0000 7f57d8cc8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d40af940 con 0x7f57d40b1730 2024-03-25T19:32:06.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.081+0000 7f57d8cc8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d40b0a70 con 0x7f57d414be90 2024-03-25T19:32:06.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.081+0000 7f57d8cc8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d40b0d50 con 0x7f57d40b4d60 2024-03-25T19:32:06.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.081+0000 7f57d8cc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:06.082 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.081+0000 7f57d8cc8640 10 monclient(hunting): authenticate will time out at 6163.441482s 2024-03-25T19:32:06.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.090+0000 7f57d259c640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f57d40b4d60 0x7f57d413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:58856/0 (socket says 172.21.15.123:58856) 2024-03-25T19:32:06.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.091+0000 7f57d259c640 1 -- 172.21.15.123:0/87307226 learned_addr learned my addr 172.21.15.123:0/87307226 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:06.096 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.096+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287212061 0 0) 0x7f57d40b0d50 con 0x7f57d40b4d60 2024-03-25T19:32:06.096 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.096+0000 7f57d159a640 10 monclient(hunting): _negotiate 2024-03-25T19:32:06.096 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.096+0000 7f57d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:06.096 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.096+0000 7f57d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:06.096 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.096+0000 7f57d159a640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:32:06.096 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.096+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57bc003520 con 0x7f57d40b4d60 2024-03-25T19:32:06.097 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.098+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231532011 0 0) 0x7f57d40b0a70 con 0x7f57d414be90 2024-03-25T19:32:06.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.098+0000 7f57d159a640 10 monclient(hunting): _negotiate 2024-03-25T19:32:06.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.098+0000 7f57d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:06.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.098+0000 7f57d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:06.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.098+0000 7f57d159a640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:32:06.098 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.098+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d40b0d50 con 0x7f57d414be90 2024-03-25T19:32:06.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.103+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752829094 0 0) 0x7f57d40af940 con 0x7f57d40b1730 2024-03-25T19:32:06.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.103+0000 7f57d159a640 10 monclient(hunting): _negotiate 2024-03-25T19:32:06.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.103+0000 7f57d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:06.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.103+0000 7f57d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:06.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.103+0000 7f57d159a640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:32:06.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.103+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d40b0a70 con 0x7f57d40b1730 2024-03-25T19:32:06.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3726941293 0 0) 0x7f57bc003520 con 0x7f57d40b4d60 2024-03-25T19:32:06.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 >> v1:172.21.15.151:6789/0 conn(0x7f57d414be90 legacy=0x7f57d414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 >> v1:172.21.15.123:6789/0 conn(0x7f57d40b1730 legacy=0x7f57d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.107 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 10 monclient: _finish_hunting 0 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 1 monclient: found mon.noname-b 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d40b1030 con 0x7f57d40b4d60 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 10 monclient: _finish_auth 0 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 10 monclient: _check_auth_tickets 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f57c4001690 con 0x7f57d40b4d60 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d159a640 10 monclient: dump: 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:06.108 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:06.109 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:06.109 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:06.109 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:06.109 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:06.109 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:06.109 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:06.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d8cc8640 5 monclient: authenticate success, global_id 4667 2024-03-25T19:32:06.109 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.107+0000 7f57d8cc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:06.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.123+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57c40017f0 con 0x7f57d40b4d60 2024-03-25T19:32:06.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.124+0000 7f57d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:06.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.124+0000 7f57d159a640 1 -- 172.21.15.123:0/87307226 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f57c4001c00 con 0x7f57d40b4d60 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.124+0000 7f57d8cc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.124+0000 7f57d8cc8640 4 set_mon_vals no callback set 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.124+0000 7f57d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.124+0000 7f57d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.124+0000 7f57d159a640 10 monclient: dump: 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:06.124 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:06.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.124+0000 7f57d8cc8640 10 monclient: shutdown 2024-03-25T19:32:06.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.124+0000 7f57d8cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:06.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.124+0000 7f57d8cc8640 1 -- 172.21.15.123:0/87307226 >> v1:172.21.15.123:6790/0 conn(0x7f57d40b4d60 legacy=0x7f57d413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.125 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.125+0000 7f57d8cc8640 1 -- 172.21.15.123:0/87307226 shutdown_connections 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.125+0000 7f57d8cc8640 1 -- 172.21.15.123:0/87307226 >> 172.21.15.123:0/87307226 conn(0x7f57d40617a0 msgr2=0x7f57d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.125+0000 7f57d8cc8640 1 -- 172.21.15.123:0/87307226 shutdown_connections 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.125+0000 7f57d8cc8640 1 -- 172.21.15.123:0/87307226 wait complete. 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.126+0000 7f57d8cc8640 10 monclient: build_initial_monmap 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.126+0000 7f57d8cc8640 10 monclient: monmap: 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:32:06.126 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:06.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.126+0000 7f57d8cc8640 1 librados: starting msgr at 2024-03-25T19:32:06.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.126+0000 7f57d8cc8640 1 librados: starting objecter 2024-03-25T19:32:06.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.126+0000 7f57d8cc8640 1 Processor -- start 2024-03-25T19:32:06.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.126+0000 7f57d8cc8640 1 -- start start 2024-03-25T19:32:06.127 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.126+0000 7f57d8cc8640 1 librados: setting wanted keys 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.126+0000 7f57d8cc8640 1 librados: calling monclient init 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.126+0000 7f57d8cc8640 10 monclient: init 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.127+0000 7f57d8cc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.127+0000 7f57d8cc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.127+0000 7f57d8cc8640 10 monclient(hunting): picked mon.noname-c con 0x7f57d40b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.127+0000 7f57d8cc8640 10 monclient(hunting): picked mon.noname-a con 0x7f57d40b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.127+0000 7f57d8cc8640 10 monclient(hunting): picked mon.noname-b con 0x7f57d414be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.127+0000 7f57d8cc8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d4140f30 con 0x7f57d40b4d60 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.127+0000 7f57d8cc8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d4140a10 con 0x7f57d414be90 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.127+0000 7f57d8cc8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d4135550 con 0x7f57d40b1730 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.127+0000 7f57d8cc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:06.128 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.127+0000 7f57d8cc8640 10 monclient(hunting): authenticate will time out at 6163.487292s 2024-03-25T19:32:06.140 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.141+0000 7f57d259c640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f57d40b1730 0x7f57d4134e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:58934/0 (socket says 172.21.15.123:58934) 2024-03-25T19:32:06.140 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.141+0000 7f57d259c640 1 -- 172.21.15.123:0/3884401492 learned_addr learned my addr 172.21.15.123:0/3884401492 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:06.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.141+0000 7f57c2ffd640 10 client.?.objecter ms_handle_connect 0x7f57d40b1730 2024-03-25T19:32:06.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.141+0000 7f57c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:06.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.144+0000 7f57c2ffd640 10 client.?.objecter ms_handle_connect 0x7f57d414be90 2024-03-25T19:32:06.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.144+0000 7f57c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:06.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.146+0000 7f57c2ffd640 10 client.?.objecter ms_handle_connect 0x7f57d40b4d60 2024-03-25T19:32:06.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.146+0000 7f57c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:06.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.146+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189209973 0 0) 0x7f57d4135550 con 0x7f57d40b1730 2024-03-25T19:32:06.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.146+0000 7f57c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:06.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.146+0000 7f57c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:06.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.146+0000 7f57c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:06.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.147+0000 7f57c2ffd640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:32:06.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.147+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b4004eb0 con 0x7f57d40b1730 2024-03-25T19:32:06.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.149+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579202907 0 0) 0x7f57d4140a10 con 0x7f57d414be90 2024-03-25T19:32:06.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.149+0000 7f57c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:06.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.149+0000 7f57c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:06.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.149+0000 7f57c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:06.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.149+0000 7f57c2ffd640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:32:06.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.149+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d4135550 con 0x7f57d414be90 2024-03-25T19:32:06.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.161+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886657922 0 0) 0x7f57d4140f30 con 0x7f57d40b4d60 2024-03-25T19:32:06.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.161+0000 7f57c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:06.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.161+0000 7f57c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:06.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.161+0000 7f57c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:06.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.161+0000 7f57c2ffd640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:32:06.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.161+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d4140a10 con 0x7f57d40b4d60 2024-03-25T19:32:06.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.164+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4072455741 0 0) 0x7f57d4135550 con 0x7f57d414be90 2024-03-25T19:32:06.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.164+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57d4140f30 con 0x7f57d414be90 2024-03-25T19:32:06.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.164+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f57cc001b50 con 0x7f57d414be90 2024-03-25T19:32:06.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.164+0000 7f57c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:06.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.164+0000 7f57c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:06.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.164+0000 7f57c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:32:06.164 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:06.164 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:06.164 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:06.164 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:06.165 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:06.165 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:06.165 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:06.165 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:06.165 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:06.165 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:06.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.164+0000 7f57c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:06.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.165+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1348161197 0 0) 0x7f57b4004eb0 con 0x7f57d40b1730 2024-03-25T19:32:06.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.165+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57d4135550 con 0x7f57d40b1730 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.165+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f57c4001bc0 con 0x7f57d40b1730 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.165+0000 7f57c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.165+0000 7f57c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.165+0000 7f57c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:06.166 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:06.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2658932560 0 0) 0x7f57d4135550 con 0x7f57d40b1730 2024-03-25T19:32:06.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 >> v1:172.21.15.151:6789/0 conn(0x7f57d414be90 legacy=0x7f57d41e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 >> v1:172.21.15.123:6789/0 conn(0x7f57d40b4d60 legacy=0x7f57d41dae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:06.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 1 monclient: found mon.c 2024-03-25T19:32:06.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:06.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:32:06.181 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d41356e0 con 0x7f57d40b1730 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57d8cc8640 5 monclient: authenticate success, global_id 4670 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57d8cc8640 10 monclient: _renew_subs 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57d8cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57d8cc8640 1 -- 172.21.15.123:0/3884401492 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d4145160 con 0x7f57d40b1730 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57d8cc8640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57d8cc8640 10 monclient: _renew_subs 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57d8cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57d8cc8640 1 -- 172.21.15.123:0/3884401492 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d4140380 con 0x7f57d40b1730 2024-03-25T19:32:06.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.181+0000 7f57d8cc8640 1 librados: init done 2024-03-25T19:32:06.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.182+0000 7f57d8cc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:06.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.182+0000 7f57d8cc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:06.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.182+0000 7f57d8cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:32:06.183 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.182+0000 7f57d8cc8640 1 -- 172.21.15.123:0/3884401492 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d4140ba0 con 0x7f57d40b1730 2024-03-25T19:32:06.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.481+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57c40011f0 con 0x7f57d40b1730 2024-03-25T19:32:06.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.481+0000 7f57c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:06.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.481+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f57c4004cb0 con 0x7f57d40b1730 2024-03-25T19:32:06.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.481+0000 7f57c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:06.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.481+0000 7f57c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:06.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.481+0000 7f57c2ffd640 10 monclient: dump: 2024-03-25T19:32:06.481 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:06.482 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:06.482 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:06.482 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:06.482 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:06.482 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:06.482 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:06.482 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:06.482 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:06.482 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:06.482 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.481+0000 7f57d0d99640 4 set_mon_vals no callback set 2024-03-25T19:32:06.483 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.482+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f57c401ad00 con 0x7f57d40b1730 2024-03-25T19:32:06.483 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.482+0000 7f57d1d9b640 10 monclient: get_auth_request con 0x7f57b406df20 auth_method 0 2024-03-25T19:32:06.483 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.483+0000 7f57c2ffd640 10 client.4670.objecter ms_handle_connect 0x7f57b406df20 2024-03-25T19:32:06.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.490+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f57c4054970 con 0x7f57d40b1730 2024-03-25T19:32:06.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.490+0000 7f57c2ffd640 10 client.4670.objecter ms_dispatch 0x7f57d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:06.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.490+0000 7f57c2ffd640 3 client.4670.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:06.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.490+0000 7f57c2ffd640 3 client.4670.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:06.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.490+0000 7f57c2ffd640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-25T19:32:06.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.490+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57c40502b0 con 0x7f57d40b1730 2024-03-25T19:32:06.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.490+0000 7f57c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:06.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.490+0000 7f57c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:06.736 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.736+0000 7f57d8cc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:06.736 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.736+0000 7f57d8cc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:06.736 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.736+0000 7f57d8cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:32:06.736 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.736+0000 7f57d8cc8640 1 -- 172.21.15.123:0/3884401492 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57d4135870 con 0x7f57d40b1730 2024-03-25T19:32:06.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.764+0000 7f57c2ffd640 1 -- 172.21.15.123:0/3884401492 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f57c40523b0 con 0x7f57d40b1730 2024-03-25T19:32:06.764 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.765+0000 7f57c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:06.765 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.765+0000 7f57c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:06.765 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:32:06.765 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:06.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 10 librados: watch_flush enter 2024-03-25T19:32:06.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 10 librados: watch_flush exit 2024-03-25T19:32:06.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-25T19:32:06.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 10 client.4670.objecter successfully canceled tick 2024-03-25T19:32:06.767 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 1 -- 172.21.15.123:0/3884401492 >> v1:172.21.15.151:6800/727684239 conn(0x7f57b406df20 legacy=0x7f57b40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 10 monclient: shutdown 2024-03-25T19:32:06.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:06.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 1 -- 172.21.15.123:0/3884401492 >> v1:172.21.15.123:6790/0 conn(0x7f57d40b1730 legacy=0x7f57d4134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 1 -- 172.21.15.123:0/3884401492 shutdown_connections 2024-03-25T19:32:06.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 1 -- 172.21.15.123:0/3884401492 >> 172.21.15.123:0/3884401492 conn(0x7f57d40617a0 msgr2=0x7f57d40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:06.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 1 -- 172.21.15.123:0/3884401492 shutdown_connections 2024-03-25T19:32:06.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 1 -- 172.21.15.123:0/3884401492 wait complete. 2024-03-25T19:32:06.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:06.767+0000 7f57d8cc8640 1 librados: shutdown 2024-03-25T19:32:06.778 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:13.583 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:13.695 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.694+0000 7f387c2a8640 10 monclient: get_monmap_and_config 2024-03-25T19:32:13.695 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.694+0000 7f387c2a8640 10 monclient: build_initial_monmap 2024-03-25T19:32:13.695 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.694+0000 7f387c2a8640 10 monclient: monmap: 2024-03-25T19:32:13.695 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:13.695 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:13.695 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:32:13.696216+0000 2024-03-25T19:32:13.695 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:32:13.696216+0000 2024-03-25T19:32:13.695 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:13.695 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:13.696 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:13.696 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:32:13.696 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:32:13.696 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:13.697 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.695+0000 7f387c2a8640 1 Processor -- start 2024-03-25T19:32:13.697 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.695+0000 7f387c2a8640 1 -- start start 2024-03-25T19:32:13.697 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.695+0000 7f387c2a8640 10 monclient: init 2024-03-25T19:32:13.697 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.696+0000 7f387c2a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:13.697 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.696+0000 7f387c2a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:13.697 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.696+0000 7f387c2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f38740a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:13.697 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.696+0000 7f387c2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f387413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:13.697 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.696+0000 7f387c2a8640 10 monclient(hunting): picked mon.noname-b con 0x7f38741430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:13.697 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.696+0000 7f387c2a8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3874142520 con 0x7f38740a98f0 2024-03-25T19:32:13.697 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.696+0000 7f387c2a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38740b28a0 con 0x7f387413f710 2024-03-25T19:32:13.698 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.696+0000 7f387c2a8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38740b2a30 con 0x7f38741430c0 2024-03-25T19:32:13.698 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.696+0000 7f387c2a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:13.698 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.696+0000 7f387c2a8640 10 monclient(hunting): authenticate will time out at 6171.057082s 2024-03-25T19:32:13.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.707+0000 7f3879843640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f387413f710 0x7f38741429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:60920/0 (socket says 172.21.15.123:60920) 2024-03-25T19:32:13.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.707+0000 7f3879843640 1 -- 172.21.15.123:0/75763750 learned_addr learned my addr 172.21.15.123:0/75763750 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:13.715 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.715+0000 7f3879042640 1 -- 172.21.15.123:0/75763750 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121314453 0 0) 0x7f38740b2a30 con 0x7f38741430c0 2024-03-25T19:32:13.716 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.715+0000 7f3879042640 10 monclient(hunting): _negotiate 2024-03-25T19:32:13.716 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.715+0000 7f3879042640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:13.716 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.715+0000 7f3879042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:13.716 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.715+0000 7f3879042640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:32:13.716 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.715+0000 7f3879042640 1 -- 172.21.15.123:0/75763750 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3860003520 con 0x7f38741430c0 2024-03-25T19:32:13.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 1 -- 172.21.15.123:0/75763750 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2957041917 0 0) 0x7f3860003520 con 0x7f38741430c0 2024-03-25T19:32:13.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 1 -- 172.21.15.123:0/75763750 >> v1:172.21.15.151:6789/0 conn(0x7f387413f710 legacy=0x7f38741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:13.725 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 1 -- 172.21.15.123:0/75763750 >> v1:172.21.15.123:6789/0 conn(0x7f38740a98f0 legacy=0x7f38740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 10 monclient: _finish_hunting 0 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 1 monclient: found mon.noname-b 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 1 -- 172.21.15.123:0/75763750 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38740b44c0 con 0x7f38741430c0 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 10 monclient: _finish_auth 0 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 10 monclient: _check_auth_tickets 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 1 -- 172.21.15.123:0/75763750 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f386c001a10 con 0x7f38741430c0 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:13.726 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f3879042640 10 monclient: dump: 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f387c2a8640 5 monclient: authenticate success, global_id 4673 2024-03-25T19:32:13.727 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.724+0000 7f387c2a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:13.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f3879042640 1 -- 172.21.15.123:0/75763750 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f386c001b70 con 0x7f38741430c0 2024-03-25T19:32:13.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f3879042640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:13.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f3879042640 1 -- 172.21.15.123:0/75763750 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f386c001f80 con 0x7f38741430c0 2024-03-25T19:32:13.731 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f387c2a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f387c2a8640 4 set_mon_vals no callback set 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f3879042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f3879042640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f3879042640 10 monclient: dump: 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:13.732 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:13.734 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f387c2a8640 10 monclient: shutdown 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f387c2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.731+0000 7f387c2a8640 1 -- 172.21.15.123:0/75763750 >> v1:172.21.15.123:6790/0 conn(0x7f38741430c0 legacy=0x7f387414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.732+0000 7f387c2a8640 1 -- 172.21.15.123:0/75763750 shutdown_connections 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.732+0000 7f387c2a8640 1 -- 172.21.15.123:0/75763750 >> 172.21.15.123:0/75763750 conn(0x7f38740b7ea0 msgr2=0x7f38740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.732+0000 7f387c2a8640 1 -- 172.21.15.123:0/75763750 shutdown_connections 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.732+0000 7f387c2a8640 1 -- 172.21.15.123:0/75763750 wait complete. 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.732+0000 7f387c2a8640 10 monclient: build_initial_monmap 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.732+0000 7f387c2a8640 10 monclient: monmap: 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:32:13.735 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 1 librados: starting msgr at 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 1 librados: starting objecter 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 1 Processor -- start 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 1 -- start start 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 1 librados: setting wanted keys 2024-03-25T19:32:13.736 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 1 librados: calling monclient init 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 10 monclient: init 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f38740a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 10 monclient(hunting): picked mon.noname-b con 0x7f387413f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f38741430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38741d7500 con 0x7f38740a98f0 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38741394a0 con 0x7f387413f710 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3874138b50 con 0x7f38741430c0 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:13.737 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.733+0000 7f387c2a8640 10 monclient(hunting): authenticate will time out at 6171.094841s 2024-03-25T19:32:13.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.754+0000 7f3879843640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f387413f710 0x7f38741d7c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:60944/0 (socket says 172.21.15.123:60944) 2024-03-25T19:32:13.754 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.754+0000 7f3879843640 1 -- 172.21.15.123:0/4190658307 learned_addr learned my addr 172.21.15.123:0/4190658307 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:13.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.755+0000 7f386affd640 10 client.?.objecter ms_handle_connect 0x7f387413f710 2024-03-25T19:32:13.756 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.755+0000 7f386affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:13.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.759+0000 7f386affd640 10 client.?.objecter ms_handle_connect 0x7f38740a98f0 2024-03-25T19:32:13.760 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.759+0000 7f386affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:13.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.761+0000 7f386affd640 10 client.?.objecter ms_handle_connect 0x7f38741430c0 2024-03-25T19:32:13.762 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.761+0000 7f386affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:13.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.767+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507815990 0 0) 0x7f38741d7500 con 0x7f38740a98f0 2024-03-25T19:32:13.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.767+0000 7f386affd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:13.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.767+0000 7f386affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:13.768 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.767+0000 7f386affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:13.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.767+0000 7f386affd640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:32:13.769 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.767+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3858004eb0 con 0x7f38740a98f0 2024-03-25T19:32:13.773 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.773+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860446242 0 0) 0x7f38741394a0 con 0x7f387413f710 2024-03-25T19:32:13.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.773+0000 7f386affd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:13.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.773+0000 7f386affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:13.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.773+0000 7f386affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:13.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.773+0000 7f386affd640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:32:13.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.773+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38741d7500 con 0x7f387413f710 2024-03-25T19:32:13.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.774+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759820400 0 0) 0x7f3874138b50 con 0x7f38741430c0 2024-03-25T19:32:13.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.774+0000 7f386affd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:13.774 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.774+0000 7f386affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:13.775 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.774+0000 7f386affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:13.775 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.774+0000 7f386affd640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:32:13.775 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.774+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38741394a0 con 0x7f38741430c0 2024-03-25T19:32:13.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.787+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1509303854 0 0) 0x7f3858004eb0 con 0x7f38740a98f0 2024-03-25T19:32:13.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.787+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3874138b50 con 0x7f38740a98f0 2024-03-25T19:32:13.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.787+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f3870001930 con 0x7f38740a98f0 2024-03-25T19:32:13.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.787+0000 7f386affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:13.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.787+0000 7f386affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:13.788 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.787+0000 7f386affd640 10 monclient(hunting): dump: 2024-03-25T19:32:13.788 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:13.788 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:13.789 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:13.789 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:13.789 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:13.789 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:13.789 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:13.789 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:13.789 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:13.789 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:13.789 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.787+0000 7f386affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:13.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.799+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1177287898 0 0) 0x7f38741394a0 con 0x7f38741430c0 2024-03-25T19:32:13.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.799+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3858004eb0 con 0x7f38741430c0 2024-03-25T19:32:13.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.799+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f386c001e80 con 0x7f38741430c0 2024-03-25T19:32:13.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.799+0000 7f386affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:13.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.799+0000 7f386affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:13.800 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.799+0000 7f386affd640 10 monclient(hunting): dump: 2024-03-25T19:32:13.800 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:13.801 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:13.801 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:13.801 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:13.801 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:13.801 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:13.801 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:13.801 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:13.801 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:13.801 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.800+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3680741740 0 0) 0x7f38741d7500 con 0x7f387413f710 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.800+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38741394a0 con 0x7f387413f710 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.800+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f38640018f0 con 0x7f387413f710 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.800+0000 7f386affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.800+0000 7f386affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.800+0000 7f386affd640 10 monclient(hunting): dump: 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:13.802 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:13.803 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:13.803 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:13.803 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:13.803 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:13.803 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:13.811 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3184092058 0 0) 0x7f3874138b50 con 0x7f38740a98f0 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 >> v1:172.21.15.123:6790/0 conn(0x7f38741430c0 legacy=0x7f3874134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 >> v1:172.21.15.151:6789/0 conn(0x7f387413f710 legacy=0x7f38741d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 1 monclient: found mon.a 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38741d8380 con 0x7f38740a98f0 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 10 monclient: _finish_auth 0 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f386affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f387c2a8640 5 monclient: authenticate success, global_id 4647 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f387c2a8640 10 monclient: _renew_subs 2024-03-25T19:32:13.812 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f387c2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:13.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.810+0000 7f387c2a8640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3874139710 con 0x7f38740a98f0 2024-03-25T19:32:13.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.811+0000 7f387c2a8640 10 client.4647.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:13.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.811+0000 7f387c2a8640 10 monclient: _renew_subs 2024-03-25T19:32:13.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.811+0000 7f387c2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:13.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.811+0000 7f387c2a8640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38741d7750 con 0x7f38740a98f0 2024-03-25T19:32:13.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.811+0000 7f387c2a8640 1 librados: init done 2024-03-25T19:32:13.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.811+0000 7f387c2a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:13.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.812+0000 7f387c2a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:13.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.812+0000 7f387c2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:13.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:13.812+0000 7f387c2a8640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38740627b0 con 0x7f38740a98f0 2024-03-25T19:32:14.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.120+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38700035f0 con 0x7f38740a98f0 2024-03-25T19:32:14.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.120+0000 7f386affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:14.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.120+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f3870003980 con 0x7f38740a98f0 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.120+0000 7f386affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.120+0000 7f386affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.120+0000 7f386affd640 10 monclient: dump: 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:14.122 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:14.123 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:14.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.120+0000 7f3878841640 4 set_mon_vals no callback set 2024-03-25T19:32:14.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.121+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f3870003ba0 con 0x7f38740a98f0 2024-03-25T19:32:14.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.122+0000 7f3879843640 10 monclient: get_auth_request con 0x7f385806e0f0 auth_method 0 2024-03-25T19:32:14.123 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.122+0000 7f386affd640 10 client.4647.objecter ms_handle_connect 0x7f385806e0f0 2024-03-25T19:32:14.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.129+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7f3870053640 con 0x7f38740a98f0 2024-03-25T19:32:14.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.129+0000 7f386affd640 10 client.4647.objecter ms_dispatch 0x7f3874000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:14.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.129+0000 7f386affd640 3 client.4647.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:14.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.129+0000 7f386affd640 3 client.4647.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:14.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.129+0000 7f386affd640 20 client.4647.objecter dump_active .. 0 homeless 2024-03-25T19:32:14.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.129+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f387004f150 con 0x7f38740a98f0 2024-03-25T19:32:14.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.129+0000 7f386affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:14.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.129+0000 7f386affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:14.353 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.352+0000 7f387c2a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:14.353 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.352+0000 7f387c2a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:14.353 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.352+0000 7f387c2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:14.353 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.352+0000 7f387c2a8640 1 -- 172.21.15.123:0/4190658307 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3874069f40 con 0x7f38740a98f0 2024-03-25T19:32:14.398 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.397+0000 7f386affd640 1 -- 172.21.15.123:0/4190658307 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7f3870051080 con 0x7f38740a98f0 2024-03-25T19:32:14.399 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.398+0000 7f386affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:14.399 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.398+0000 7f386affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:14.399 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:32:14.399 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:14.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.400+0000 7f387c2a8640 10 librados: watch_flush enter 2024-03-25T19:32:14.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.400+0000 7f387c2a8640 10 librados: watch_flush exit 2024-03-25T19:32:14.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.400+0000 7f387c2a8640 20 client.4647.objecter shutdown clearing up homeless session... 2024-03-25T19:32:14.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.400+0000 7f387c2a8640 10 client.4647.objecter successfully canceled tick 2024-03-25T19:32:14.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.401+0000 7f387c2a8640 1 -- 172.21.15.123:0/4190658307 >> v1:172.21.15.151:6800/727684239 conn(0x7f385806e0f0 legacy=0x7f38580705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.401+0000 7f387c2a8640 10 monclient: shutdown 2024-03-25T19:32:14.402 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.401+0000 7f387c2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:14.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.401+0000 7f387c2a8640 1 -- 172.21.15.123:0/4190658307 >> v1:172.21.15.123:6789/0 conn(0x7f38740a98f0 legacy=0x7f3874138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.401+0000 7f387c2a8640 1 -- 172.21.15.123:0/4190658307 shutdown_connections 2024-03-25T19:32:14.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.401+0000 7f387c2a8640 1 -- 172.21.15.123:0/4190658307 >> 172.21.15.123:0/4190658307 conn(0x7f38740b7ea0 msgr2=0x7f387413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:14.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.401+0000 7f387c2a8640 1 -- 172.21.15.123:0/4190658307 shutdown_connections 2024-03-25T19:32:14.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.401+0000 7f387c2a8640 1 -- 172.21.15.123:0/4190658307 wait complete. 2024-03-25T19:32:14.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:14.401+0000 7f387c2a8640 1 librados: shutdown 2024-03-25T19:32:14.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:16.635 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:10:06.208 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:16.641 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:10:06.214 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:16.688 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:10:06.260 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:16.695 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:10:06.228 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:16.700 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:10:06.273 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:16.705 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:10:06.239 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:16.713 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:10:06.286 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:16.730 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:10:06.263 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:16.739 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:10:06.311 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:21.220 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:21.338 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.338+0000 7fe7e02f4640 10 monclient: get_monmap_and_config 2024-03-25T19:32:21.338 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.338+0000 7fe7e02f4640 10 monclient: build_initial_monmap 2024-03-25T19:32:21.338 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.338+0000 7fe7e02f4640 10 monclient: monmap: 2024-03-25T19:32:21.338 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:21.338 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:21.338 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:32:21.339146+0000 2024-03-25T19:32:21.338 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:32:21.339146+0000 2024-03-25T19:32:21.338 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:21.338 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:21.339 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:21.339 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:32:21.339 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:32:21.339 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:21.339 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.339+0000 7fe7e02f4640 1 Processor -- start 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.339+0000 7fe7e02f4640 1 -- start start 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.339+0000 7fe7e02f4640 10 monclient: init 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.340+0000 7fe7e02f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.340+0000 7fe7e02f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.340+0000 7fe7e02f4640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d80a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.340+0000 7fe7e02f4640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.340+0000 7fe7e02f4640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d81430c0 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.340+0000 7fe7e02f4640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d8142520 con 0x7fe7d80a98f0 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.340+0000 7fe7e02f4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80b28a0 con 0x7fe7d813f710 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.340+0000 7fe7e02f4640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80b2a30 con 0x7fe7d81430c0 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.340+0000 7fe7e02f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:21.340 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.340+0000 7fe7e02f4640 10 monclient(hunting): authenticate will time out at 6178.700250s 2024-03-25T19:32:21.347 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.347+0000 7fe7dd88f640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fe7d813f710 0x7fe7d81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:57138/0 (socket says 172.21.15.123:57138) 2024-03-25T19:32:21.347 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.347+0000 7fe7dd88f640 1 -- 172.21.15.123:0/526538578 learned_addr learned my addr 172.21.15.123:0/526538578 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:21.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.357+0000 7fe7dd08e640 1 -- 172.21.15.123:0/526538578 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460283247 0 0) 0x7fe7d80b28a0 con 0x7fe7d813f710 2024-03-25T19:32:21.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.357+0000 7fe7dd08e640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.357+0000 7fe7dd08e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.357+0000 7fe7dd08e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.357+0000 7fe7dd08e640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:32:21.357 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.357+0000 7fe7dd08e640 1 -- 172.21.15.123:0/526538578 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c4003520 con 0x7fe7d813f710 2024-03-25T19:32:21.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 1 -- 172.21.15.123:0/526538578 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2551886919 0 0) 0x7fe7c4003520 con 0x7fe7d813f710 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 1 -- 172.21.15.123:0/526538578 >> v1:172.21.15.123:6790/0 conn(0x7fe7d81430c0 legacy=0x7fe7d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 1 -- 172.21.15.123:0/526538578 >> v1:172.21.15.123:6789/0 conn(0x7fe7d80a98f0 legacy=0x7fe7d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 10 monclient: _finish_hunting 0 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 1 monclient: found mon.noname-c 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 1 -- 172.21.15.123:0/526538578 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d80b44c0 con 0x7fe7d813f710 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 10 monclient: _finish_auth 0 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 10 monclient: _check_auth_tickets 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 1 -- 172.21.15.123:0/526538578 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe7c8001690 con 0x7fe7d813f710 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.370+0000 7fe7dd08e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.371+0000 7fe7dd08e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.371+0000 7fe7dd08e640 10 monclient: dump: 2024-03-25T19:32:21.371 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.371+0000 7fe7e02f4640 5 monclient: authenticate success, global_id 4660 2024-03-25T19:32:21.372 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.371+0000 7fe7e02f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:21.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.391+0000 7fe7dd08e640 1 -- 172.21.15.123:0/526538578 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7c80017f0 con 0x7fe7d813f710 2024-03-25T19:32:21.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.391+0000 7fe7dd08e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:21.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.391+0000 7fe7dd08e640 1 -- 172.21.15.123:0/526538578 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe7c8001c00 con 0x7fe7d813f710 2024-03-25T19:32:21.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.391+0000 7fe7dd08e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.391+0000 7fe7dd08e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:21.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.391+0000 7fe7dd08e640 10 monclient: dump: 2024-03-25T19:32:21.391 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.391+0000 7fe7e02f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:21.392 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.391+0000 7fe7e02f4640 4 set_mon_vals no callback set 2024-03-25T19:32:21.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.392+0000 7fe7e02f4640 10 monclient: shutdown 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.392+0000 7fe7e02f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.392+0000 7fe7e02f4640 1 -- 172.21.15.123:0/526538578 >> v1:172.21.15.151:6789/0 conn(0x7fe7d813f710 legacy=0x7fe7d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.392+0000 7fe7e02f4640 1 -- 172.21.15.123:0/526538578 shutdown_connections 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.392+0000 7fe7e02f4640 1 -- 172.21.15.123:0/526538578 >> 172.21.15.123:0/526538578 conn(0x7fe7d80b7ea0 msgr2=0x7fe7d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.392+0000 7fe7e02f4640 1 -- 172.21.15.123:0/526538578 shutdown_connections 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.392+0000 7fe7e02f4640 1 -- 172.21.15.123:0/526538578 wait complete. 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.393+0000 7fe7e02f4640 10 monclient: build_initial_monmap 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.393+0000 7fe7e02f4640 10 monclient: monmap: 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:21.395 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.393+0000 7fe7e02f4640 1 librados: starting msgr at 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.393+0000 7fe7e02f4640 1 librados: starting objecter 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.393+0000 7fe7e02f4640 1 Processor -- start 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.393+0000 7fe7e02f4640 1 -- start start 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.393+0000 7fe7e02f4640 1 librados: setting wanted keys 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.393+0000 7fe7e02f4640 1 librados: calling monclient init 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.393+0000 7fe7e02f4640 10 monclient: init 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.394+0000 7fe7e02f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.394+0000 7fe7e02f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:32:21.396 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.394+0000 7fe7e02f4640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d80a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:21.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.394+0000 7fe7e02f4640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d813f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:21.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.394+0000 7fe7e02f4640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d81430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:21.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.394+0000 7fe7e02f4640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80b1870 con 0x7fe7d80a98f0 2024-03-25T19:32:21.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.394+0000 7fe7e02f4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80ac910 con 0x7fe7d81430c0 2024-03-25T19:32:21.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.394+0000 7fe7e02f4640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80abb00 con 0x7fe7d813f710 2024-03-25T19:32:21.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.394+0000 7fe7e02f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:21.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.394+0000 7fe7e02f4640 10 monclient(hunting): authenticate will time out at 6178.754564s 2024-03-25T19:32:21.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.398+0000 7fe7de891640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fe7d81430c0 0x7fe7d81e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:57166/0 (socket says 172.21.15.123:57166) 2024-03-25T19:32:21.397 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.398+0000 7fe7de891640 1 -- 172.21.15.123:0/2313605971 learned_addr learned my addr 172.21.15.123:0/2313605971 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:21.399 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.400+0000 7fe7ceffd640 10 client.?.objecter ms_handle_connect 0x7fe7d81430c0 2024-03-25T19:32:21.399 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.400+0000 7fe7ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:21.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.403+0000 7fe7ceffd640 10 client.?.objecter ms_handle_connect 0x7fe7d80a98f0 2024-03-25T19:32:21.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.403+0000 7fe7ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:21.411 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.412+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676810165 0 0) 0x7fe7d80ac910 con 0x7fe7d81430c0 2024-03-25T19:32:21.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.412+0000 7fe7ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.412+0000 7fe7ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.412+0000 7fe7ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.412+0000 7fe7ceffd640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:32:21.412 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.412+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7bc004eb0 con 0x7fe7d81430c0 2024-03-25T19:32:21.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.414+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244536545 0 0) 0x7fe7d8142520 con 0x7fe7d80a98f0 2024-03-25T19:32:21.413 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.414+0000 7fe7ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.414+0000 7fe7ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.414+0000 7fe7ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.414+0000 7fe7ceffd640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:32:21.414 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.414+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7bc0051d0 con 0x7fe7d80a98f0 2024-03-25T19:32:21.423 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.423+0000 7fe7ceffd640 10 client.?.objecter ms_handle_connect 0x7fe7d813f710 2024-03-25T19:32:21.423 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.423+0000 7fe7ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:21.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.431+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908184139 0 0) 0x7fe7d80abb00 con 0x7fe7d813f710 2024-03-25T19:32:21.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.431+0000 7fe7ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.431+0000 7fe7ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.431+0000 7fe7ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.431+0000 7fe7ceffd640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:32:21.431 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.431+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d8142520 con 0x7fe7d813f710 2024-03-25T19:32:21.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.441+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 600918515 0 0) 0x7fe7bc004eb0 con 0x7fe7d81430c0 2024-03-25T19:32:21.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.441+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7d80abb00 con 0x7fe7d81430c0 2024-03-25T19:32:21.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.441+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe7d0001c50 con 0x7fe7d81430c0 2024-03-25T19:32:21.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.441+0000 7fe7ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.441+0000 7fe7ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:21.441 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.441+0000 7fe7ceffd640 10 monclient(hunting): dump: 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:21.442 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.441+0000 7fe7ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.446+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 42592015 0 0) 0x7fe7bc0051d0 con 0x7fe7d80a98f0 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.446+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7bc004eb0 con 0x7fe7d80a98f0 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.446+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe7d4001790 con 0x7fe7d80a98f0 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.446+0000 7fe7ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.446+0000 7fe7ceffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.446+0000 7fe7ceffd640 10 monclient(hunting): dump: 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:21.446 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.447 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:21.447 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:21.447 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:21.447 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:21.447 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:21.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3367016093 0 0) 0x7fe7d80abb00 con 0x7fe7d81430c0 2024-03-25T19:32:21.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 >> v1:172.21.15.123:6790/0 conn(0x7fe7d813f710 legacy=0x7fe7d80ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 >> v1:172.21.15.123:6789/0 conn(0x7fe7d80a98f0 legacy=0x7fe7d8138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:21.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 1 monclient: found mon.b 2024-03-25T19:32:21.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:21.456 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d80abde0 con 0x7fe7d81430c0 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 10 monclient: _finish_auth 0 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7e02f4640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7e02f4640 10 monclient: _renew_subs 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7e02f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7e02f4640 1 -- 172.21.15.123:0/2313605971 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d8138f40 con 0x7fe7d81430c0 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7e02f4640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7e02f4640 10 monclient: _renew_subs 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7e02f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7e02f4640 1 -- 172.21.15.123:0/2313605971 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d81363f0 con 0x7fe7d81430c0 2024-03-25T19:32:21.457 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.456+0000 7fe7e02f4640 1 librados: init done 2024-03-25T19:32:21.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.457+0000 7fe7e02f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:21.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.457+0000 7fe7e02f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:21.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.457+0000 7fe7e02f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:21.458 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.457+0000 7fe7e02f4640 1 -- 172.21.15.123:0/2313605971 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d80627b0 con 0x7fe7d81430c0 2024-03-25T19:32:21.784 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.784+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7d0003990 con 0x7fe7d81430c0 2024-03-25T19:32:21.784 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.784+0000 7fe7ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:21.784 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.784+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fe7d0003d20 con 0x7fe7d81430c0 2024-03-25T19:32:21.784 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.784+0000 7fe7ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.784 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.784+0000 7fe7ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.784+0000 7fe7ceffd640 10 monclient: dump: 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:21.785 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.784+0000 7fe7dc88d640 4 set_mon_vals no callback set 2024-03-25T19:32:21.786 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.785+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fe7d0003f00 con 0x7fe7d81430c0 2024-03-25T19:32:21.786 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.786+0000 7fe7de090640 10 monclient: get_auth_request con 0x7fe7bc06e130 auth_method 0 2024-03-25T19:32:21.786 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.786+0000 7fe7ceffd640 10 client.4663.objecter ms_handle_connect 0x7fe7bc06e130 2024-03-25T19:32:21.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.790+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fe7d0054ae0 con 0x7fe7d81430c0 2024-03-25T19:32:21.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.790+0000 7fe7ceffd640 10 client.4663.objecter ms_dispatch 0x7fe7d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:21.790 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.790+0000 7fe7ceffd640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:21.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.790+0000 7fe7ceffd640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:21.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.790+0000 7fe7ceffd640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T19:32:21.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.790+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7d00505f0 con 0x7fe7d81430c0 2024-03-25T19:32:21.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.790+0000 7fe7ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:21.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:21.790+0000 7fe7ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:22.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.033+0000 7fe7e02f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:22.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.033+0000 7fe7e02f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:22.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.033+0000 7fe7e02f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:22.033 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.033+0000 7fe7e02f4640 1 -- 172.21.15.123:0/2313605971 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7d81434a0 con 0x7fe7d81430c0 2024-03-25T19:32:22.071 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.072+0000 7fe7ceffd640 1 -- 172.21.15.123:0/2313605971 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fe7d0052520 con 0x7fe7d81430c0 2024-03-25T19:32:22.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.072+0000 7fe7ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:22.072 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.072+0000 7fe7ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:22.072 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:32:22.072 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 10 librados: watch_flush enter 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 10 librados: watch_flush exit 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 10 client.4663.objecter successfully canceled tick 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 1 -- 172.21.15.123:0/2313605971 >> v1:172.21.15.151:6800/727684239 conn(0x7fe7bc06e130 legacy=0x7fe7bc0705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 10 monclient: shutdown 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 1 -- 172.21.15.123:0/2313605971 >> v1:172.21.15.151:6789/0 conn(0x7fe7d81430c0 legacy=0x7fe7d81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 1 -- 172.21.15.123:0/2313605971 shutdown_connections 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 1 -- 172.21.15.123:0/2313605971 >> 172.21.15.123:0/2313605971 conn(0x7fe7d80b7ea0 msgr2=0x7fe7d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:22.075 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 1 -- 172.21.15.123:0/2313605971 shutdown_connections 2024-03-25T19:32:22.076 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.074+0000 7fe7e02f4640 1 -- 172.21.15.123:0/2313605971 wait complete. 2024-03-25T19:32:22.076 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:22.075+0000 7fe7e02f4640 1 librados: shutdown 2024-03-25T19:32:22.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:28.892 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:28.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.990+0000 7fbdfb5b6640 10 monclient: get_monmap_and_config 2024-03-25T19:32:28.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.990+0000 7fbdfb5b6640 10 monclient: build_initial_monmap 2024-03-25T19:32:28.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.991+0000 7fbdfb5b6640 10 monclient: monmap: 2024-03-25T19:32:28.991 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:28.991 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:28.991 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:32:28.991916+0000 2024-03-25T19:32:28.991 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:32:28.991916+0000 2024-03-25T19:32:28.991 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:28.991 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:28.991 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:28.991 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:32:28.991 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:32:28.991 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:28.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.991+0000 7fbdfb5b6640 1 Processor -- start 2024-03-25T19:32:28.992 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.991+0000 7fbdfb5b6640 1 -- start start 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.991+0000 7fbdfb5b6640 10 monclient: init 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.992+0000 7fbdfb5b6640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.992+0000 7fbdfb5b6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.992+0000 7fbdfb5b6640 10 monclient(hunting): picked mon.noname-a con 0x7fbdf40a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.992+0000 7fbdfb5b6640 10 monclient(hunting): picked mon.noname-b con 0x7fbdf413f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.992+0000 7fbdfb5b6640 10 monclient(hunting): picked mon.noname-c con 0x7fbdf41430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.992+0000 7fbdfb5b6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf4142520 con 0x7fbdf40a98f0 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.992+0000 7fbdfb5b6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40b28a0 con 0x7fbdf41430c0 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.992+0000 7fbdfb5b6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40b2a30 con 0x7fbdf413f710 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.992+0000 7fbdfb5b6640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:28.993 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:28.992+0000 7fbdfb5b6640 10 monclient(hunting): authenticate will time out at 6186.352482s 2024-03-25T19:32:29.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.005+0000 7fbdf9352640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fbdf40a98f0 0x7fbdf40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:60316/0 (socket says 172.21.15.123:60316) 2024-03-25T19:32:29.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.005+0000 7fbdf9352640 1 -- 172.21.15.123:0/657201739 learned_addr learned my addr 172.21.15.123:0/657201739 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:29.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.016+0000 7fbdebfff640 1 -- 172.21.15.123:0/657201739 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743012367 0 0) 0x7fbdf4142520 con 0x7fbdf40a98f0 2024-03-25T19:32:29.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.016+0000 7fbdebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:29.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.016+0000 7fbdebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:29.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.016+0000 7fbdebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:29.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.016+0000 7fbdebfff640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:32:29.016 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.016+0000 7fbdebfff640 1 -- 172.21.15.123:0/657201739 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde0003520 con 0x7fbdf40a98f0 2024-03-25T19:32:29.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 1 -- 172.21.15.123:0/657201739 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3554847307 0 0) 0x7fbde0003520 con 0x7fbdf40a98f0 2024-03-25T19:32:29.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 1 -- 172.21.15.123:0/657201739 >> v1:172.21.15.123:6790/0 conn(0x7fbdf413f710 legacy=0x7fbdf41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 1 -- 172.21.15.123:0/657201739 >> v1:172.21.15.151:6789/0 conn(0x7fbdf41430c0 legacy=0x7fbdf414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:29.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 1 monclient: found mon.noname-a 2024-03-25T19:32:29.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:29.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:32:29.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 1 -- 172.21.15.123:0/657201739 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40b44c0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 10 monclient: _finish_auth 0 2024-03-25T19:32:29.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 1 -- 172.21.15.123:0/657201739 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbdf0001690 con 0x7fbdf40a98f0 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdebfff640 10 monclient: dump: 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:29.090 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdfb5b6640 5 monclient: authenticate success, global_id 4656 2024-03-25T19:32:29.091 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.089+0000 7fbdfb5b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:29.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.111+0000 7fbdebfff640 1 -- 172.21.15.123:0/657201739 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdf00017f0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.111+0000 7fbdebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:29.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.111+0000 7fbdebfff640 1 -- 172.21.15.123:0/657201739 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbdf0001c00 con 0x7fbdf40a98f0 2024-03-25T19:32:29.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.111+0000 7fbdebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.111+0000 7fbdebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:29.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.111+0000 7fbdebfff640 10 monclient: dump: 2024-03-25T19:32:29.111 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:29.111 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:29.112 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:29.112 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:29.112 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.112 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:29.112 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:29.112 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:29.112 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:29.112 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:29.112 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.111+0000 7fbdfb5b6640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:29.112 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.111+0000 7fbdfb5b6640 4 set_mon_vals no callback set 2024-03-25T19:32:29.113 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.112+0000 7fbdfb5b6640 10 monclient: shutdown 2024-03-25T19:32:29.113 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.112+0000 7fbdfb5b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.112+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/657201739 >> v1:172.21.15.123:6789/0 conn(0x7fbdf40a98f0 legacy=0x7fbdf40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.112+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/657201739 shutdown_connections 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.112+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/657201739 >> 172.21.15.123:0/657201739 conn(0x7fbdf40b7ea0 msgr2=0x7fbdf40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.112+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/657201739 shutdown_connections 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.112+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/657201739 wait complete. 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.113+0000 7fbdfb5b6640 10 monclient: build_initial_monmap 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.113+0000 7fbdfb5b6640 10 monclient: monmap: 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:29.114 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.113+0000 7fbdfb5b6640 1 librados: starting msgr at 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.113+0000 7fbdfb5b6640 1 librados: starting objecter 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.114+0000 7fbdfb5b6640 1 Processor -- start 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.114+0000 7fbdfb5b6640 1 -- start start 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.114+0000 7fbdfb5b6640 1 librados: setting wanted keys 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.114+0000 7fbdfb5b6640 1 librados: calling monclient init 2024-03-25T19:32:29.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.114+0000 7fbdfb5b6640 10 monclient: init 2024-03-25T19:32:29.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.114+0000 7fbdfb5b6640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:29.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.114+0000 7fbdfb5b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:29.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.115+0000 7fbdfb5b6640 10 monclient(hunting): picked mon.noname-b con 0x7fbdf40a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:29.116 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.115+0000 7fbdfb5b6640 10 monclient(hunting): picked mon.noname-c con 0x7fbdf413f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:29.117 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.115+0000 7fbdfb5b6640 10 monclient(hunting): picked mon.noname-a con 0x7fbdf41430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:29.117 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.115+0000 7fbdfb5b6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf4135ac0 con 0x7fbdf41430c0 2024-03-25T19:32:29.117 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.115+0000 7fbdfb5b6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf4137030 con 0x7fbdf40a98f0 2024-03-25T19:32:29.117 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.115+0000 7fbdfb5b6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf41372a0 con 0x7fbdf413f710 2024-03-25T19:32:29.117 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.115+0000 7fbdfb5b6640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:29.117 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.115+0000 7fbdfb5b6640 10 monclient(hunting): authenticate will time out at 6186.475026s 2024-03-25T19:32:29.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.121+0000 7fbdf9352640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fbdf40a98f0 0x7fbdf4133440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:38134/0 (socket says 172.21.15.123:38134) 2024-03-25T19:32:29.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.121+0000 7fbdf9352640 1 -- 172.21.15.123:0/3172724923 learned_addr learned my addr 172.21.15.123:0/3172724923 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:29.121 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.122+0000 7fbde9ffb640 10 client.?.objecter ms_handle_connect 0x7fbdf40a98f0 2024-03-25T19:32:29.122 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.122+0000 7fbde9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:29.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.130+0000 7fbde9ffb640 10 client.?.objecter ms_handle_connect 0x7fbdf41430c0 2024-03-25T19:32:29.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.130+0000 7fbde9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:29.134 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.134+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022847772 0 0) 0x7fbdf4135ac0 con 0x7fbdf41430c0 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.134+0000 7fbde9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.134+0000 7fbde9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.134+0000 7fbde9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.134+0000 7fbde9ffb640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.134+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8004eb0 con 0x7fbdf41430c0 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.135+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480006143 0 0) 0x7fbdf4137030 con 0x7fbdf40a98f0 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.135+0000 7fbde9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.135+0000 7fbde9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.135+0000 7fbde9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.135+0000 7fbde9ffb640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:32:29.135 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.135+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf4135ac0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.149+0000 7fbde9ffb640 10 client.?.objecter ms_handle_connect 0x7fbdf413f710 2024-03-25T19:32:29.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.149+0000 7fbde9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:29.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.151+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3930751650 0 0) 0x7fbdd8004eb0 con 0x7fbdf41430c0 2024-03-25T19:32:29.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.151+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdf4137030 con 0x7fbdf41430c0 2024-03-25T19:32:29.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.151+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbdec001b90 con 0x7fbdf41430c0 2024-03-25T19:32:29.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.151+0000 7fbde9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.151+0000 7fbde9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:29.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.151+0000 7fbde9ffb640 10 monclient(hunting): dump: 2024-03-25T19:32:29.151 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:29.152 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:29.152 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:29.152 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:29.152 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.152 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:29.152 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:29.152 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:29.152 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:29.152 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:29.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.151+0000 7fbde9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.152+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1048810016 0 0) 0x7fbdf4135ac0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.152+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdd8004eb0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.152+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbdf0001c00 con 0x7fbdf40a98f0 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.152+0000 7fbde9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.152+0000 7fbde9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.152+0000 7fbde9ffb640 10 monclient(hunting): dump: 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:29.153 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:29.154 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:29.154 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:29.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.160+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856843547 0 0) 0x7fbdf41372a0 con 0x7fbdf413f710 2024-03-25T19:32:29.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.160+0000 7fbde9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:29.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.160+0000 7fbde9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:29.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.160+0000 7fbde9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:29.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.160+0000 7fbde9ffb640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:32:29.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.160+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf4135ac0 con 0x7fbdf413f710 2024-03-25T19:32:29.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 308934596 0 0) 0x7fbdd8004eb0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 >> v1:172.21.15.123:6790/0 conn(0x7fbdf413f710 legacy=0x7fbdf41379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 >> v1:172.21.15.123:6789/0 conn(0x7fbdf41430c0 legacy=0x7fbdf41e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:29.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 1 monclient: found mon.b 2024-03-25T19:32:29.171 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf4137580 con 0x7fbdf40a98f0 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbde9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbdfb5b6640 5 monclient: authenticate success, global_id 4669 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbdfb5b6640 10 monclient: _renew_subs 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbdfb5b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/3172724923 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf41363b0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbdfb5b6640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbdfb5b6640 10 monclient: _renew_subs 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbdfb5b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/3172724923 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf41353a0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.172 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.171+0000 7fbdfb5b6640 1 librados: init done 2024-03-25T19:32:29.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.173+0000 7fbdfb5b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:29.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.173+0000 7fbdfb5b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:29.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.173+0000 7fbdfb5b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:29.173 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.173+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/3172724923 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf40627b0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.462+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdf0004a90 con 0x7fbdf40a98f0 2024-03-25T19:32:29.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.462+0000 7fbde9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:29.462 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.462+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbdf0004da0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.462+0000 7fbde9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.462+0000 7fbde9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.462+0000 7fbde9ffb640 10 monclient: dump: 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:29.463 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.462+0000 7fbdeb7fe640 4 set_mon_vals no callback set 2024-03-25T19:32:29.464 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.463+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fbdf001ad70 con 0x7fbdf40a98f0 2024-03-25T19:32:29.464 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.464+0000 7fbdf8b51640 10 monclient: get_auth_request con 0x7fbdd806df50 auth_method 0 2024-03-25T19:32:29.464 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.464+0000 7fbde9ffb640 10 client.4669.objecter ms_handle_connect 0x7fbdd806df50 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.468+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3707419625 0 0) 0x7fbdf0054930 con 0x7fbdf40a98f0 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.468+0000 7fbde9ffb640 10 client.4669.objecter ms_dispatch 0x7fbdf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.468+0000 7fbde9ffb640 3 client.4669.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.468+0000 7fbde9ffb640 3 client.4669.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.468+0000 7fbde9ffb640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T19:32:29.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.469+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbdf0050440 con 0x7fbdf40a98f0 2024-03-25T19:32:29.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.469+0000 7fbde9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:29.469 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.469+0000 7fbde9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:29.717 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.718+0000 7fbdfb5b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:29.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.718+0000 7fbdfb5b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:29.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.718+0000 7fbdfb5b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:29.718 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.718+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/3172724923 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdf4074cc0 con 0x7fbdf40a98f0 2024-03-25T19:32:29.773 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.773+0000 7fbde9ffb640 1 -- 172.21.15.123:0/3172724923 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2206167630) 0x7fbdf0052370 con 0x7fbdf40a98f0 2024-03-25T19:32:29.773 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.773+0000 7fbde9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:29.773 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.773+0000 7fbde9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:29.774 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:32:29.774 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:22:38.023202+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:22:38.023202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:29.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.775+0000 7fbdfb5b6640 10 librados: watch_flush enter 2024-03-25T19:32:29.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.775+0000 7fbdfb5b6640 10 librados: watch_flush exit 2024-03-25T19:32:29.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.775+0000 7fbdfb5b6640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T19:32:29.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.775+0000 7fbdfb5b6640 10 client.4669.objecter successfully canceled tick 2024-03-25T19:32:29.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.776+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/3172724923 >> v1:172.21.15.151:6800/727684239 conn(0x7fbdd806df50 legacy=0x7fbdd8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.776+0000 7fbdfb5b6640 10 monclient: shutdown 2024-03-25T19:32:29.776 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.776+0000 7fbdfb5b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:29.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.776+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/3172724923 >> v1:172.21.15.151:6789/0 conn(0x7fbdf40a98f0 legacy=0x7fbdf4133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.776+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/3172724923 shutdown_connections 2024-03-25T19:32:29.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.776+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/3172724923 >> 172.21.15.123:0/3172724923 conn(0x7fbdf40b7ea0 msgr2=0x7fbdf4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:29.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.776+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/3172724923 shutdown_connections 2024-03-25T19:32:29.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.776+0000 7fbdfb5b6640 1 -- 172.21.15.123:0/3172724923 wait complete. 2024-03-25T19:32:29.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:29.776+0000 7fbdfb5b6640 1 librados: shutdown 2024-03-25T19:32:29.788 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:36.595 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:36.707 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.708+0000 7f7734e55640 10 monclient: get_monmap_and_config 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.708+0000 7f7734e55640 10 monclient: build_initial_monmap 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.708+0000 7f7734e55640 10 monclient: monmap: 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:32:36.709105+0000 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:32:36.709105+0000 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:32:36.708 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:36.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.708+0000 7f7734e55640 1 Processor -- start 2024-03-25T19:32:36.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.708+0000 7f7734e55640 1 -- start start 2024-03-25T19:32:36.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.708+0000 7f7734e55640 10 monclient: init 2024-03-25T19:32:36.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.709+0000 7f7734e55640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:36.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.709+0000 7f7734e55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:36.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.709+0000 7f7734e55640 10 monclient(hunting): picked mon.noname-c con 0x7f77300b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:36.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.709+0000 7f7734e55640 10 monclient(hunting): picked mon.noname-b con 0x7f77300b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:36.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.709+0000 7f7734e55640 10 monclient(hunting): picked mon.noname-a con 0x7f773014be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:36.710 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.709+0000 7f7734e55640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77300af940 con 0x7f773014be90 2024-03-25T19:32:36.711 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.709+0000 7f7734e55640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77300b0a70 con 0x7f77300b4d60 2024-03-25T19:32:36.711 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.709+0000 7f7734e55640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77300b0d50 con 0x7f77300b1730 2024-03-25T19:32:36.711 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.709+0000 7f7734e55640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:36.711 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.709+0000 7f7734e55640 10 monclient(hunting): authenticate will time out at 6194.069491s 2024-03-25T19:32:36.711 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.710+0000 7f772dd9b640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f77300b1730 0x7f77300b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:39594/0 (socket says 172.21.15.123:39594) 2024-03-25T19:32:36.711 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.710+0000 7f772dd9b640 1 -- 172.21.15.123:0/2458106999 learned_addr learned my addr 172.21.15.123:0/2458106999 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:36.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.782+0000 7f772d59a640 1 -- 172.21.15.123:0/2458106999 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577623490 0 0) 0x7f77300af940 con 0x7f773014be90 2024-03-25T19:32:36.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.782+0000 7f772d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:32:36.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.782+0000 7f772d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:36.782 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.782+0000 7f772d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:36.783 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.782+0000 7f772d59a640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:32:36.783 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.782+0000 7f772d59a640 1 -- 172.21.15.123:0/2458106999 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7718003520 con 0x7f773014be90 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 1 -- 172.21.15.123:0/2458106999 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 674472463 0 0) 0x7f7718003520 con 0x7f773014be90 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 1 -- 172.21.15.123:0/2458106999 >> v1:172.21.15.123:6790/0 conn(0x7f77300b1730 legacy=0x7f77300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 1 -- 172.21.15.123:0/2458106999 >> v1:172.21.15.151:6789/0 conn(0x7f77300b4d60 legacy=0x7f773013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 1 monclient: found mon.noname-a 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 1 -- 172.21.15.123:0/2458106999 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77300b1030 con 0x7f773014be90 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 10 monclient: _finish_auth 0 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f772d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:36.797 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f7734e55640 5 monclient: authenticate success, global_id 4662 2024-03-25T19:32:36.798 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.797+0000 7f7734e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.801+0000 7f772d59a640 1 -- 172.21.15.123:0/2458106999 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7720001690 con 0x7f773014be90 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.801+0000 7f772d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.801+0000 7f772d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.801+0000 7f772d59a640 10 monclient: dump: 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:36.801 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:36.802 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:36.802 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:36.802 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.801+0000 7f7734e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:36.813 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.812+0000 7f772d59a640 1 -- 172.21.15.123:0/2458106999 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7720001920 con 0x7f773014be90 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.812+0000 7f772d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.812+0000 7f772d59a640 1 -- 172.21.15.123:0/2458106999 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7720001c30 con 0x7f773014be90 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.812+0000 7f7734e55640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f7734e55640 4 set_mon_vals no callback set 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f772d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f772d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f772d59a640 10 monclient: dump: 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:36.814 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f7734e55640 10 monclient: shutdown 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f7734e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f7734e55640 1 -- 172.21.15.123:0/2458106999 >> v1:172.21.15.123:6789/0 conn(0x7f773014be90 legacy=0x7f773014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f7734e55640 1 -- 172.21.15.123:0/2458106999 shutdown_connections 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f7734e55640 1 -- 172.21.15.123:0/2458106999 >> 172.21.15.123:0/2458106999 conn(0x7f77300617a0 msgr2=0x7f7730061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f7734e55640 1 -- 172.21.15.123:0/2458106999 shutdown_connections 2024-03-25T19:32:36.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.813+0000 7f7734e55640 1 -- 172.21.15.123:0/2458106999 wait complete. 2024-03-25T19:32:36.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.814+0000 7f7734e55640 10 monclient: build_initial_monmap 2024-03-25T19:32:36.817 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.814+0000 7f7734e55640 10 monclient: monmap: 2024-03-25T19:32:36.817 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:36.817 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:36.817 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:32:36.817 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.814+0000 7f7734e55640 1 librados: starting msgr at 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.814+0000 7f7734e55640 1 librados: starting objecter 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.814+0000 7f7734e55640 1 Processor -- start 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 1 -- start start 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 1 librados: setting wanted keys 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 1 librados: calling monclient init 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 10 monclient: init 2024-03-25T19:32:36.818 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 10 monclient(hunting): picked mon.noname-c con 0x7f77300b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 10 monclient(hunting): picked mon.noname-b con 0x7f77300b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 10 monclient(hunting): picked mon.noname-a con 0x7f773014be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77301405b0 con 0x7f773014be90 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77301356f0 con 0x7f77300b4d60 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7730137a90 con 0x7f77300b1730 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.815+0000 7f7734e55640 10 monclient(hunting): authenticate will time out at 6194.175480s 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.816+0000 7f772dd9b640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f77300b4d60 0x7f77301dac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:55964/0 (socket says 172.21.15.123:55964) 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.816+0000 7f772dd9b640 1 -- 172.21.15.123:0/887357426 learned_addr learned my addr 172.21.15.123:0/887357426 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.816+0000 7f771effd640 10 client.?.objecter ms_handle_connect 0x7f77300b4d60 2024-03-25T19:32:36.819 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.816+0000 7f771effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:36.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.826+0000 7f771effd640 10 client.?.objecter ms_handle_connect 0x7f773014be90 2024-03-25T19:32:36.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.826+0000 7f771effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:36.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.832+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450326083 0 0) 0x7f77301405b0 con 0x7f773014be90 2024-03-25T19:32:36.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.832+0000 7f771effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:36.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.832+0000 7f771effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:36.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.832+0000 7f771effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:36.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.832+0000 7f771effd640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:32:36.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.832+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7710004eb0 con 0x7f773014be90 2024-03-25T19:32:36.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.841+0000 7f771effd640 10 client.?.objecter ms_handle_connect 0x7f77300b1730 2024-03-25T19:32:36.840 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.841+0000 7f771effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:36.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.846+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4106091873 0 0) 0x7f7710004eb0 con 0x7f773014be90 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.846+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77301405b0 con 0x7f773014be90 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.846+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7720001650 con 0x7f773014be90 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.846+0000 7f771effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.846+0000 7f771effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.846+0000 7f771effd640 10 monclient(hunting): dump: 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:36.847 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:36.848 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:36.848 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:36.848 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.847+0000 7f771effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:36.861 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:10:26.433 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:36.866 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:10:26.439 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:36.867 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:10:26.440 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:36.869 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:10:26.441 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:36.874 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:10:26.408 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:36.876 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:10:26.410 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:36.930 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:10:26.502 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:36.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.942+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1006596237 0 0) 0x7f77301405b0 con 0x7f773014be90 2024-03-25T19:32:36.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.942+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 >> v1:172.21.15.123:6790/0 conn(0x7f77300b1730 legacy=0x7f7730134fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.942+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 >> v1:172.21.15.151:6789/0 conn(0x7f77300b4d60 legacy=0x7f77301dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f771effd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:36.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f771effd640 1 monclient: found mon.a 2024-03-25T19:32:36.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f771effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:36.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f771effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:36.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7730137c20 con 0x7f773014be90 2024-03-25T19:32:36.943 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f771effd640 10 monclient: _finish_auth 0 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f771effd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f771effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f7734e55640 5 monclient: authenticate success, global_id 4665 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f7734e55640 10 monclient: _renew_subs 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f7734e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f7734e55640 1 -- 172.21.15.123:0/887357426 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77301454a0 con 0x7f773014be90 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f7734e55640 10 client.4665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f7734e55640 10 monclient: _renew_subs 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f7734e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f7734e55640 1 -- 172.21.15.123:0/887357426 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7730138480 con 0x7f773014be90 2024-03-25T19:32:36.944 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.943+0000 7f7734e55640 1 librados: init done 2024-03-25T19:32:36.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.944+0000 7f7734e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:36.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.944+0000 7f7734e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:36.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.944+0000 7f7734e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:36.945 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:36.944+0000 7f7734e55640 1 -- 172.21.15.123:0/887357426 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7730140760 con 0x7f773014be90 2024-03-25T19:32:36.980 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:10:26.514 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:37.155 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.156+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77200038a0 con 0x7f773014be90 2024-03-25T19:32:37.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.156+0000 7f771effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:37.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.156+0000 7f772cd99640 4 set_mon_vals no callback set 2024-03-25T19:32:37.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.196+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f7720003a10 con 0x7f773014be90 2024-03-25T19:32:37.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.196+0000 7f771effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:37.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.196+0000 7f771effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:37.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.196+0000 7f771effd640 10 monclient: dump: 2024-03-25T19:32:37.196 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:37.196 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:37.196 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:37.196 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:37.197 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:37.197 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:37.197 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:37.197 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:37.197 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:37.197 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:37.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.197+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f77200199e0 con 0x7f773014be90 2024-03-25T19:32:37.198 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:10:26.769 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:37.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.198+0000 7f772e59c640 10 monclient: get_auth_request con 0x7f771006dc90 auth_method 0 2024-03-25T19:32:37.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.198+0000 7f771effd640 10 client.4665.objecter ms_handle_connect 0x7f771006dc90 2024-03-25T19:32:37.204 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.204+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7f7720053460 con 0x7f773014be90 2024-03-25T19:32:37.204 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.204+0000 7f771effd640 10 client.4665.objecter ms_dispatch 0x7f7730000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:32:37.204 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.204+0000 7f771effd640 3 client.4665.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:32:37.204 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.204+0000 7f771effd640 3 client.4665.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:32:37.204 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.204+0000 7f771effd640 20 client.4665.objecter dump_active .. 0 homeless 2024-03-25T19:32:37.204 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.204+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f772004ef70 con 0x7f773014be90 2024-03-25T19:32:37.204 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.204+0000 7f771effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:37.204 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.204+0000 7f771effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:37.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.390+0000 7f7734e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:37.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.390+0000 7f7734e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:37.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.390+0000 7f7734e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:37.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.390+0000 7f7734e55640 1 -- 172.21.15.123:0/887357426 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77300627b0 con 0x7f773014be90 2024-03-25T19:32:37.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.418+0000 7f771effd640 1 -- 172.21.15.123:0/887357426 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7f7720050ea0 con 0x7f773014be90 2024-03-25T19:32:37.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.418+0000 7f771effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:37.418 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.418+0000 7f771effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:37.418 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:32:37.418 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:37.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 10 librados: watch_flush enter 2024-03-25T19:32:37.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 10 librados: watch_flush exit 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 20 client.4665.objecter shutdown clearing up homeless session... 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 10 client.4665.objecter successfully canceled tick 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 1 -- 172.21.15.123:0/887357426 >> v1:172.21.15.151:6800/727684239 conn(0x7f771006dc90 legacy=0x7f7710070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 10 monclient: shutdown 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 1 -- 172.21.15.123:0/887357426 >> v1:172.21.15.123:6789/0 conn(0x7f773014be90 legacy=0x7f77301db340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 1 -- 172.21.15.123:0/887357426 shutdown_connections 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 1 -- 172.21.15.123:0/887357426 >> 172.21.15.123:0/887357426 conn(0x7f77300617a0 msgr2=0x7f773013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 1 -- 172.21.15.123:0/887357426 shutdown_connections 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 1 -- 172.21.15.123:0/887357426 wait complete. 2024-03-25T19:32:37.422 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:37.421+0000 7f7734e55640 1 librados: shutdown 2024-03-25T19:32:37.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:44.238 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:44.348 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.349+0000 7f5420b4d640 10 monclient: get_monmap_and_config 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.349+0000 7f5420b4d640 10 monclient: build_initial_monmap 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.349+0000 7f5420b4d640 10 monclient: monmap: 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:32:44.350042+0000 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:32:44.350042+0000 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:32:44.349 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:44.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.350+0000 7f5420b4d640 1 Processor -- start 2024-03-25T19:32:44.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.350+0000 7f5420b4d640 1 -- start start 2024-03-25T19:32:44.350 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.350+0000 7f5420b4d640 10 monclient: init 2024-03-25T19:32:44.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.351+0000 7f5420b4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:44.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.351+0000 7f5420b4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:44.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.351+0000 7f5420b4d640 10 monclient(hunting): picked mon.noname-b con 0x7f541c0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:44.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.351+0000 7f5420b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f541c0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:44.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.351+0000 7f5420b4d640 10 monclient(hunting): picked mon.noname-a con 0x7f541c14be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:44.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.351+0000 7f5420b4d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0af940 con 0x7f541c14be90 2024-03-25T19:32:44.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.351+0000 7f5420b4d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0b0a70 con 0x7f541c0b1730 2024-03-25T19:32:44.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.351+0000 7f5420b4d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0b0d50 con 0x7f541c0b4d60 2024-03-25T19:32:44.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.351+0000 7f5420b4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:44.351 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.351+0000 7f5420b4d640 10 monclient(hunting): authenticate will time out at 6201.711108s 2024-03-25T19:32:44.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.354+0000 7f541ad9d640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f541c14be90 0x7f541c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:38606/0 (socket says 172.21.15.123:38606) 2024-03-25T19:32:44.354 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.354+0000 7f541ad9d640 1 -- 172.21.15.123:0/3752211900 learned_addr learned my addr 172.21.15.123:0/3752211900 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:44.369 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084413031 0 0) 0x7f541c0af940 con 0x7f541c14be90 2024-03-25T19:32:44.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:32:44.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:44.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:44.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:32:44.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5408003520 con 0x7f541c14be90 2024-03-25T19:32:44.370 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261565884 0 0) 0x7f541c0b0a70 con 0x7f541c0b1730 2024-03-25T19:32:44.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:32:44.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:44.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:44.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:32:44.371 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.370+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f541c0af940 con 0x7f541c0b1730 2024-03-25T19:32:44.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2549334285 0 0) 0x7f541c0af940 con 0x7f541c0b1730 2024-03-25T19:32:44.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 >> v1:172.21.15.123:6790/0 conn(0x7f541c0b4d60 legacy=0x7f541c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 >> v1:172.21.15.123:6789/0 conn(0x7f541c14be90 legacy=0x7f541c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:32:44.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 1 monclient: found mon.noname-c 2024-03-25T19:32:44.379 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c0b1030 con 0x7f541c0b1730 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 10 monclient: _finish_auth 0 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f5404001670 con 0x7f541c0b1730 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5420b4d640 5 monclient: authenticate success, global_id 4675 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5420b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5419d9b640 10 monclient: dump: 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:44.380 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:44.381 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:44.381 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:44.381 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:44.381 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:44.381 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:44.381 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:44.381 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:44.381 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.379+0000 7f5420b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:44.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.388+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54040017d0 con 0x7f541c0b1730 2024-03-25T19:32:44.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.388+0000 7f5419d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:44.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.388+0000 7f5419d9b640 1 -- 172.21.15.123:0/3752211900 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f5404001ba0 con 0x7f541c0b1730 2024-03-25T19:32:44.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.388+0000 7f5419d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:44.388 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.388+0000 7f5419d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.388+0000 7f5419d9b640 10 monclient: dump: 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.388+0000 7f5420b4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:44.389 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.388+0000 7f5420b4d640 4 set_mon_vals no callback set 2024-03-25T19:32:44.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.389+0000 7f5420b4d640 10 monclient: shutdown 2024-03-25T19:32:44.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.389+0000 7f5420b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:44.390 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.389+0000 7f5420b4d640 1 -- 172.21.15.123:0/3752211900 >> v1:172.21.15.151:6789/0 conn(0x7f541c0b1730 legacy=0x7f541c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.390+0000 7f5420b4d640 1 -- 172.21.15.123:0/3752211900 shutdown_connections 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.390+0000 7f5420b4d640 1 -- 172.21.15.123:0/3752211900 >> 172.21.15.123:0/3752211900 conn(0x7f541c0617a0 msgr2=0x7f541c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.390+0000 7f5420b4d640 1 -- 172.21.15.123:0/3752211900 shutdown_connections 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.390+0000 7f5420b4d640 1 -- 172.21.15.123:0/3752211900 wait complete. 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.391+0000 7f5420b4d640 10 monclient: build_initial_monmap 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.391+0000 7f5420b4d640 10 monclient: monmap: 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:32:44.391 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:32:44.392 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:44.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.391+0000 7f5420b4d640 1 librados: starting msgr at 2024-03-25T19:32:44.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.391+0000 7f5420b4d640 1 librados: starting objecter 2024-03-25T19:32:44.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.391+0000 7f5420b4d640 1 Processor -- start 2024-03-25T19:32:44.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.391+0000 7f5420b4d640 1 -- start start 2024-03-25T19:32:44.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.391+0000 7f5420b4d640 1 librados: setting wanted keys 2024-03-25T19:32:44.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.391+0000 7f5420b4d640 1 librados: calling monclient init 2024-03-25T19:32:44.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.391+0000 7f5420b4d640 10 monclient: init 2024-03-25T19:32:44.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.392+0000 7f5420b4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:44.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.392+0000 7f5420b4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:44.393 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.392+0000 7f5420b4d640 10 monclient(hunting): picked mon.noname-a con 0x7f541c0b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:44.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.392+0000 7f5420b4d640 10 monclient(hunting): picked mon.noname-b con 0x7f541c0b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:44.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.392+0000 7f5420b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f541c14be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:44.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.392+0000 7f5420b4d640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c1d7850 con 0x7f541c0b1730 2024-03-25T19:32:44.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.392+0000 7f5420b4d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c1382f0 con 0x7f541c0b4d60 2024-03-25T19:32:44.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.392+0000 7f5420b4d640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c1d83f0 con 0x7f541c14be90 2024-03-25T19:32:44.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.392+0000 7f5420b4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:44.394 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.392+0000 7f5420b4d640 10 monclient(hunting): authenticate will time out at 6201.752430s 2024-03-25T19:32:44.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.403+0000 7f541ad9d640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f541c14be90 0x7f541c12fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:44136/0 (socket says 172.21.15.123:44136) 2024-03-25T19:32:44.403 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.403+0000 7f541ad9d640 1 -- 172.21.15.123:0/2812039569 learned_addr learned my addr 172.21.15.123:0/2812039569 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:44.404 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.405+0000 7f5412ffd640 10 client.?.objecter ms_handle_connect 0x7f541c0b1730 2024-03-25T19:32:44.404 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.405+0000 7f5412ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:44.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.405+0000 7f5412ffd640 10 client.?.objecter ms_handle_connect 0x7f541c14be90 2024-03-25T19:32:44.405 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.405+0000 7f5412ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:44.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.416+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364515445 0 0) 0x7f541c1d7850 con 0x7f541c0b1730 2024-03-25T19:32:44.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.416+0000 7f5412ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:44.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.416+0000 7f5412ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:44.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.416+0000 7f5412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:44.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.416+0000 7f5412ffd640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:32:44.416 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.416+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53fc004eb0 con 0x7f541c0b1730 2024-03-25T19:32:44.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.417+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540357180 0 0) 0x7f541c1d83f0 con 0x7f541c14be90 2024-03-25T19:32:44.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.417+0000 7f5412ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:44.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.417+0000 7f5412ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:44.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.417+0000 7f5412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:44.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.417+0000 7f5412ffd640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:32:44.417 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.417+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f541c1d7850 con 0x7f541c14be90 2024-03-25T19:32:44.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.422+0000 7f5412ffd640 10 client.?.objecter ms_handle_connect 0x7f541c0b4d60 2024-03-25T19:32:44.421 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.422+0000 7f5412ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:44.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.430+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 158693476 0 0) 0x7f541c1d7850 con 0x7f541c14be90 2024-03-25T19:32:44.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.430+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f541c1d83f0 con 0x7f541c14be90 2024-03-25T19:32:44.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.430+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f5414001b30 con 0x7f541c14be90 2024-03-25T19:32:44.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.430+0000 7f5412ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:44.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.430+0000 7f5412ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:44.430 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.430+0000 7f5412ffd640 10 monclient(hunting): dump: 2024-03-25T19:32:44.430 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:44.430 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:44.430 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:44.430 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:44.431 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:44.431 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:44.431 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:44.431 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:44.431 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:44.431 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:44.431 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.430+0000 7f5412ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.434+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425005875 0 0) 0x7f53fc004eb0 con 0x7f541c0b1730 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.434+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f541c1d7850 con 0x7f541c0b1730 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.434+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f540c001930 con 0x7f541c0b1730 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.434+0000 7f5412ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.434+0000 7f5412ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.434+0000 7f5412ffd640 10 monclient(hunting): dump: 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:44.434 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:44.435 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:44.435 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:44.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2967118893 0 0) 0x7f541c1d83f0 con 0x7f541c14be90 2024-03-25T19:32:44.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 >> v1:172.21.15.151:6789/0 conn(0x7f541c0b4d60 legacy=0x7f541c136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 >> v1:172.21.15.123:6789/0 conn(0x7f541c0b1730 legacy=0x7f541c1d7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:44.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 1 monclient: found mon.c 2024-03-25T19:32:44.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:44.438 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c135c90 con 0x7f541c14be90 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 10 monclient: _finish_auth 0 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5412ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5420b4d640 5 monclient: authenticate success, global_id 4694 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5420b4d640 10 monclient: _renew_subs 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5420b4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5420b4d640 1 -- 172.21.15.123:0/2812039569 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c136e80 con 0x7f541c14be90 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5420b4d640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5420b4d640 10 monclient: _renew_subs 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5420b4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5420b4d640 1 -- 172.21.15.123:0/2812039569 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c143ae0 con 0x7f541c14be90 2024-03-25T19:32:44.439 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.438+0000 7f5420b4d640 1 librados: init done 2024-03-25T19:32:44.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.439+0000 7f5420b4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:44.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.439+0000 7f5420b4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:44.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.439+0000 7f5420b4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:32:44.440 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.439+0000 7f5420b4d640 1 -- 172.21.15.123:0/2812039569 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541c0627b0 con 0x7f541c14be90 2024-03-25T19:32:44.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.443+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5414001fa0 con 0x7f541c14be90 2024-03-25T19:32:44.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.443+0000 7f5412ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:44.443 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.443+0000 7f541959a640 4 set_mon_vals no callback set 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.581+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f5414004d00 con 0x7f541c14be90 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.581+0000 7f5412ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.581+0000 7f5412ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.581+0000 7f5412ffd640 10 monclient: dump: 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:44.581 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:44.582 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:44.582 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:44.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.582+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f5414004ee0 con 0x7f541c14be90 2024-03-25T19:32:44.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.582+0000 7f541a59c640 10 monclient: get_auth_request con 0x7f53fc06de90 auth_method 0 2024-03-25T19:32:44.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.583+0000 7f5412ffd640 10 client.4694.objecter ms_handle_connect 0x7f53fc06de90 2024-03-25T19:32:44.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.744+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7f54140549d0 con 0x7f541c14be90 2024-03-25T19:32:44.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.744+0000 7f5412ffd640 10 client.4694.objecter ms_dispatch 0x7f541c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:32:44.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.744+0000 7f5412ffd640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:32:44.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.744+0000 7f5412ffd640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:32:44.744 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.744+0000 7f5412ffd640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T19:32:44.748 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.749+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54140504e0 con 0x7f541c14be90 2024-03-25T19:32:44.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.749+0000 7f5412ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:44.749 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.749+0000 7f5412ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:44.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.968+0000 7f5420b4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:44.968 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.968+0000 7f5420b4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:44.968 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.968+0000 7f5420b4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:32:44.968 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.968+0000 7f5420b4d640 1 -- 172.21.15.123:0/2812039569 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f541c1d7a00 con 0x7f541c14be90 2024-03-25T19:32:44.995 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.995+0000 7f5412ffd640 1 -- 172.21.15.123:0/2812039569 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7f541c1d7a00 con 0x7f541c14be90 2024-03-25T19:32:44.995 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.996+0000 7f5412ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:44.995 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.996+0000 7f5412ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:44.996 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:32:44.996 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:44.998 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 10 librados: watch_flush enter 2024-03-25T19:32:44.998 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 10 librados: watch_flush exit 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 10 client.4694.objecter successfully canceled tick 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 1 -- 172.21.15.123:0/2812039569 >> v1:172.21.15.151:6800/727684239 conn(0x7f53fc06de90 legacy=0x7f53fc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 10 monclient: shutdown 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 1 -- 172.21.15.123:0/2812039569 >> v1:172.21.15.123:6790/0 conn(0x7f541c14be90 legacy=0x7f541c12fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 1 -- 172.21.15.123:0/2812039569 shutdown_connections 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 1 -- 172.21.15.123:0/2812039569 >> 172.21.15.123:0/2812039569 conn(0x7f541c0617a0 msgr2=0x7f541c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 1 -- 172.21.15.123:0/2812039569 shutdown_connections 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 1 -- 172.21.15.123:0/2812039569 wait complete. 2024-03-25T19:32:44.999 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:44.998+0000 7f5420b4d640 1 librados: shutdown 2024-03-25T19:32:45.009 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:51.816 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:51.914 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.914+0000 7f646198c640 10 monclient: get_monmap_and_config 2024-03-25T19:32:51.914 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.914+0000 7f646198c640 10 monclient: build_initial_monmap 2024-03-25T19:32:51.914 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.914+0000 7f646198c640 10 monclient: monmap: 2024-03-25T19:32:51.914 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:51.914 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:51.915 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:32:51.915710+0000 2024-03-25T19:32:51.915 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:32:51.915710+0000 2024-03-25T19:32:51.915 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:51.915 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:51.915 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:51.915 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:32:51.915 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:32:51.915 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:51.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.915+0000 7f646198c640 1 Processor -- start 2024-03-25T19:32:51.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.915+0000 7f646198c640 1 -- start start 2024-03-25T19:32:51.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.915+0000 7f646198c640 10 monclient: init 2024-03-25T19:32:51.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.916+0000 7f646198c640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:51.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.916+0000 7f646198c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:51.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.916+0000 7f646198c640 10 monclient(hunting): picked mon.noname-b con 0x7f645c13f990 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:51.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.916+0000 7f646198c640 10 monclient(hunting): picked mon.noname-a con 0x7f645c13fd70 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:51.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.916+0000 7f646198c640 10 monclient(hunting): picked mon.noname-c con 0x7f645c0afa60 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:51.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.916+0000 7f646198c640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c0afeb0 con 0x7f645c13fd70 2024-03-25T19:32:51.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.916+0000 7f646198c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c0b0100 con 0x7f645c0afa60 2024-03-25T19:32:51.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.916+0000 7f646198c640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c0b1150 con 0x7f645c13f990 2024-03-25T19:32:51.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.916+0000 7f646198c640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:51.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.916+0000 7f646198c640 10 monclient(hunting): authenticate will time out at 6209.276080s 2024-03-25T19:32:51.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.924+0000 7f645b7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f645c0afa60 0x7f645c14dbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:43718/0 (socket says 172.21.15.123:43718) 2024-03-25T19:32:51.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.924+0000 7f645b7fe640 1 -- 172.21.15.123:0/2435697749 learned_addr learned my addr 172.21.15.123:0/2435697749 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.938+0000 7f6459ffb640 1 -- 172.21.15.123:0/2435697749 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177077199 0 0) 0x7f645c0b0100 con 0x7f645c0afa60 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.938+0000 7f6459ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.938+0000 7f6459ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.938+0000 7f6459ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.938+0000 7f6459ffb640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.938+0000 7f6459ffb640 1 -- 172.21.15.123:0/2435697749 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6444003520 con 0x7f645c0afa60 2024-03-25T19:32:51.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 1 -- 172.21.15.123:0/2435697749 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 782006397 0 0) 0x7f6444003520 con 0x7f645c0afa60 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 1 -- 172.21.15.123:0/2435697749 >> v1:172.21.15.123:6790/0 conn(0x7f645c13f990 legacy=0x7f645c06ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 1 -- 172.21.15.123:0/2435697749 >> v1:172.21.15.123:6789/0 conn(0x7f645c13fd70 legacy=0x7f645c145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 1 monclient: found mon.noname-c 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 1 -- 172.21.15.123:0/2435697749 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c0aede0 con 0x7f645c0afa60 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 1 -- 172.21.15.123:0/2435697749 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6454001a10 con 0x7f645c0afa60 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:51.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f6459ffb640 10 monclient: dump: 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f646198c640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:32:51.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.948+0000 7f646198c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:51.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f6459ffb640 1 -- 172.21.15.123:0/2435697749 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6454001be0 con 0x7f645c0afa60 2024-03-25T19:32:51.951 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f6459ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f6459ffb640 1 -- 172.21.15.123:0/2435697749 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6454002030 con 0x7f645c0afa60 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f6459ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f6459ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f6459ffb640 10 monclient: dump: 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:51.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f646198c640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:51.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f646198c640 4 set_mon_vals no callback set 2024-03-25T19:32:51.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f646198c640 10 monclient: shutdown 2024-03-25T19:32:51.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f646198c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:51.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.952+0000 7f646198c640 1 -- 172.21.15.123:0/2435697749 >> v1:172.21.15.151:6789/0 conn(0x7f645c0afa60 legacy=0x7f645c14dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:51.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.953+0000 7f646198c640 1 -- 172.21.15.123:0/2435697749 shutdown_connections 2024-03-25T19:32:51.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.953+0000 7f646198c640 1 -- 172.21.15.123:0/2435697749 >> 172.21.15.123:0/2435697749 conn(0x7f645c0b4b80 msgr2=0x7f645c0b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:51.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.953+0000 7f646198c640 1 -- 172.21.15.123:0/2435697749 shutdown_connections 2024-03-25T19:32:51.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.953+0000 7f646198c640 1 -- 172.21.15.123:0/2435697749 wait complete. 2024-03-25T19:32:51.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.953+0000 7f646198c640 10 monclient: build_initial_monmap 2024-03-25T19:32:51.954 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.953+0000 7f646198c640 10 monclient: monmap: 2024-03-25T19:32:51.954 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.954+0000 7f646198c640 1 librados: starting msgr at 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.954+0000 7f646198c640 1 librados: starting objecter 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.954+0000 7f646198c640 1 Processor -- start 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.954+0000 7f646198c640 1 -- start start 2024-03-25T19:32:51.955 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.954+0000 7f646198c640 1 librados: setting wanted keys 2024-03-25T19:32:51.956 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.954+0000 7f646198c640 1 librados: calling monclient init 2024-03-25T19:32:51.956 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.954+0000 7f646198c640 10 monclient: init 2024-03-25T19:32:51.956 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.955+0000 7f646198c640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:51.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.955+0000 7f646198c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:51.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.955+0000 7f646198c640 10 monclient(hunting): picked mon.noname-c con 0x7f645c0afa60 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:51.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.955+0000 7f646198c640 10 monclient(hunting): picked mon.noname-b con 0x7f645c13f990 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:51.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.955+0000 7f646198c640 10 monclient(hunting): picked mon.noname-a con 0x7f645c13fd70 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:51.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.955+0000 7f646198c640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c11f370 con 0x7f645c13fd70 2024-03-25T19:32:51.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.955+0000 7f646198c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c121510 con 0x7f645c13f990 2024-03-25T19:32:51.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.955+0000 7f646198c640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c1206e0 con 0x7f645c0afa60 2024-03-25T19:32:51.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.955+0000 7f646198c640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:51.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.955+0000 7f646198c640 10 monclient(hunting): authenticate will time out at 6209.315470s 2024-03-25T19:32:51.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.966+0000 7f645b7fe640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f645c13fd70 0x7f645c1e24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:42704/0 (socket says 172.21.15.123:42704) 2024-03-25T19:32:51.965 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.966+0000 7f645b7fe640 1 -- 172.21.15.123:0/4062290721 learned_addr learned my addr 172.21.15.123:0/4062290721 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:51.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.968+0000 7f6443fff640 10 client.?.objecter ms_handle_connect 0x7f645c13fd70 2024-03-25T19:32:51.967 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.968+0000 7f6443fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:51.971 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.972+0000 7f6443fff640 10 client.?.objecter ms_handle_connect 0x7f645c13f990 2024-03-25T19:32:51.971 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.972+0000 7f6443fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:51.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.977+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431508095 0 0) 0x7f645c11f370 con 0x7f645c13fd70 2024-03-25T19:32:51.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.977+0000 7f6443fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:51.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.977+0000 7f6443fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:51.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.977+0000 7f6443fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:51.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.977+0000 7f6443fff640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:32:51.977 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.977+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6448004eb0 con 0x7f645c13fd70 2024-03-25T19:32:51.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.983+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877970909 0 0) 0x7f645c121510 con 0x7f645c13f990 2024-03-25T19:32:51.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.983+0000 7f6443fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:51.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.983+0000 7f6443fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:51.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.983+0000 7f6443fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:51.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.983+0000 7f6443fff640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:32:51.983 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.983+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f645c11f370 con 0x7f645c13f990 2024-03-25T19:32:51.988 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.988+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3766982816 0 0) 0x7f6448004eb0 con 0x7f645c13fd70 2024-03-25T19:32:51.988 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.989+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f645c121510 con 0x7f645c13fd70 2024-03-25T19:32:51.988 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.989+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6454001ef0 con 0x7f645c13fd70 2024-03-25T19:32:51.988 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.989+0000 7f6443fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.989+0000 7f6443fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.989+0000 7f6443fff640 10 monclient(hunting): dump: 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:51.989 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.989+0000 7f6443fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:51.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.989+0000 7f6443fff640 10 client.?.objecter ms_handle_connect 0x7f645c0afa60 2024-03-25T19:32:51.990 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:51.989+0000 7f6443fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:52.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.002+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023081458 0 0) 0x7f645c1206e0 con 0x7f645c0afa60 2024-03-25T19:32:52.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.002+0000 7f6443fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:52.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.002+0000 7f6443fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:52.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.002+0000 7f6443fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:52.001 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.002+0000 7f6443fff640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:32:52.002 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.002+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6448004eb0 con 0x7f645c0afa60 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.005+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3293512624 0 0) 0x7f645c11f370 con 0x7f645c13f990 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.005+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f645c1206e0 con 0x7f645c13f990 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.005+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6450001790 con 0x7f645c13f990 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.005+0000 7f6443fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.005+0000 7f6443fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.005+0000 7f6443fff640 10 monclient(hunting): dump: 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:52.005 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:52.006 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:52.006 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:52.006 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:52.006 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:52.006 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:52.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1401336228 0 0) 0x7f645c121510 con 0x7f645c13fd70 2024-03-25T19:32:52.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 >> v1:172.21.15.123:6790/0 conn(0x7f645c0afa60 legacy=0x7f645c11ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 >> v1:172.21.15.151:6789/0 conn(0x7f645c13f990 legacy=0x7f645c1d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:52.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 1 monclient: found mon.a 2024-03-25T19:32:52.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:52.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:52.007 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c1d8190 con 0x7f645c13fd70 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 10 monclient: _finish_auth 0 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f6443fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f646198c640 5 monclient: authenticate success, global_id 4674 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f646198c640 10 monclient: _renew_subs 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f646198c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f646198c640 1 -- 172.21.15.123:0/4062290721 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645c13df20 con 0x7f645c13fd70 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f646198c640 10 client.4674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f646198c640 10 monclient: _renew_subs 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f646198c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f646198c640 1 -- 172.21.15.123:0/4062290721 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f645c11f800 con 0x7f645c13fd70 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.006+0000 7f646198c640 1 librados: init done 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.007+0000 7f646198c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.007+0000 7f646198c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.007+0000 7f646198c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:52.008 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.007+0000 7f646198c640 1 -- 172.21.15.123:0/4062290721 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645c145f30 con 0x7f645c13fd70 2024-03-25T19:32:52.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.304+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6454004f70 con 0x7f645c13fd70 2024-03-25T19:32:52.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.304+0000 7f6443fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:52.304 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.304+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f6454005280 con 0x7f645c13fd70 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.304+0000 7f6443fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.304+0000 7f6443fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.304+0000 7f6443fff640 10 monclient: dump: 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:52.305 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.304+0000 7f64597fa640 4 set_mon_vals no callback set 2024-03-25T19:32:52.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.305+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f645401b250 con 0x7f645c13fd70 2024-03-25T19:32:52.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.306+0000 7f645affd640 10 monclient: get_auth_request con 0x7f644806df10 auth_method 0 2024-03-25T19:32:52.306 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.306+0000 7f6443fff640 10 client.4674.objecter ms_handle_connect 0x7f644806df10 2024-03-25T19:32:52.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.313+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7f6454054ce0 con 0x7f645c13fd70 2024-03-25T19:32:52.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.313+0000 7f6443fff640 10 client.4674.objecter ms_dispatch 0x7f645c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:32:52.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.313+0000 7f6443fff640 3 client.4674.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:32:52.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.313+0000 7f6443fff640 3 client.4674.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:32:52.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.313+0000 7f6443fff640 20 client.4674.objecter dump_active .. 0 homeless 2024-03-25T19:32:52.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.313+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64540507f0 con 0x7f645c13fd70 2024-03-25T19:32:52.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.313+0000 7f6443fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:52.313 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.313+0000 7f6443fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:52.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.569+0000 7f646198c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:52.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.569+0000 7f646198c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:52.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.569+0000 7f646198c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:32:52.569 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.569+0000 7f646198c640 1 -- 172.21.15.123:0/4062290721 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f645c0627b0 con 0x7f645c13fd70 2024-03-25T19:32:52.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.599+0000 7f6443fff640 1 -- 172.21.15.123:0/4062290721 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7f6454052720 con 0x7f645c13fd70 2024-03-25T19:32:52.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.600+0000 7f6443fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:52.599 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.600+0000 7f6443fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:52.600 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:32:52.600 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:52.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.602+0000 7f646198c640 10 librados: watch_flush enter 2024-03-25T19:32:52.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.602+0000 7f646198c640 10 librados: watch_flush exit 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.602+0000 7f646198c640 20 client.4674.objecter shutdown clearing up homeless session... 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.602+0000 7f646198c640 10 client.4674.objecter successfully canceled tick 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.602+0000 7f646198c640 1 -- 172.21.15.123:0/4062290721 >> v1:172.21.15.151:6800/727684239 conn(0x7f644806df10 legacy=0x7f64480703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.602+0000 7f646198c640 10 monclient: shutdown 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.602+0000 7f646198c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.602+0000 7f646198c640 1 -- 172.21.15.123:0/4062290721 >> v1:172.21.15.123:6789/0 conn(0x7f645c13fd70 legacy=0x7f645c1e24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.603+0000 7f646198c640 1 -- 172.21.15.123:0/4062290721 shutdown_connections 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.603+0000 7f646198c640 1 -- 172.21.15.123:0/4062290721 >> 172.21.15.123:0/4062290721 conn(0x7f645c0b4b80 msgr2=0x7f645c13f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.603+0000 7f646198c640 1 -- 172.21.15.123:0/4062290721 shutdown_connections 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.603+0000 7f646198c640 1 -- 172.21.15.123:0/4062290721 wait complete. 2024-03-25T19:32:52.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:52.603+0000 7f646198c640 1 librados: shutdown 2024-03-25T19:32:52.614 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:56.704 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:10:46.276 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:56.760 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:10:46.332 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:56.791 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:10:46.363 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:56.814 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:10:46.348 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:56.819 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:10:46.392 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:56.820 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:10:46.354 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:56.823 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:10:46.395 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:56.842 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:10:46.376 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:56.850 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:10:46.423 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:59.420 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:59.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.536+0000 7f8295cb3640 10 monclient: get_monmap_and_config 2024-03-25T19:32:59.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.536+0000 7f8295cb3640 10 monclient: build_initial_monmap 2024-03-25T19:32:59.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.536+0000 7f8295cb3640 10 monclient: monmap: 2024-03-25T19:32:59.537 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:59.537 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:59.537 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:32:59.538517+0000 2024-03-25T19:32:59.537 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:32:59.538517+0000 2024-03-25T19:32:59.537 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:59.538 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:59.538 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:59.538 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:32:59.538 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:32:59.538 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:59.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.537+0000 7f8295cb3640 1 Processor -- start 2024-03-25T19:32:59.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.537+0000 7f8295cb3640 1 -- start start 2024-03-25T19:32:59.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.537+0000 7f8295cb3640 10 monclient: init 2024-03-25T19:32:59.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.538+0000 7f8295cb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:59.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.538+0000 7f8295cb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:59.539 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.538+0000 7f8295cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f82900b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:59.540 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.538+0000 7f8295cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f82900b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:59.540 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.538+0000 7f8295cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f829014be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:59.540 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.538+0000 7f8295cb3640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82900af940 con 0x7f829014be90 2024-03-25T19:32:59.540 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.538+0000 7f8295cb3640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82900b0a70 con 0x7f82900b4d60 2024-03-25T19:32:59.540 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.538+0000 7f8295cb3640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82900b0d50 con 0x7f82900b1730 2024-03-25T19:32:59.540 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.538+0000 7f8295cb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:59.540 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.538+0000 7f8295cb3640 10 monclient(hunting): authenticate will time out at 6216.899301s 2024-03-25T19:32:59.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.540+0000 7f828effd640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f82900b1730 0x7f82900b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:36792/0 (socket says 172.21.15.123:36792) 2024-03-25T19:32:59.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.540+0000 7f828effd640 1 -- 172.21.15.123:0/1091694696 learned_addr learned my addr 172.21.15.123:0/1091694696 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:59.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.555+0000 7f828e7fc640 1 -- 172.21.15.123:0/1091694696 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484579799 0 0) 0x7f82900b0d50 con 0x7f82900b1730 2024-03-25T19:32:59.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.555+0000 7f828e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:59.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.555+0000 7f828e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:59.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.555+0000 7f828e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:59.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.555+0000 7f828e7fc640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:32:59.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.555+0000 7f828e7fc640 1 -- 172.21.15.123:0/1091694696 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8278003520 con 0x7f82900b1730 2024-03-25T19:32:59.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 1 -- 172.21.15.123:0/1091694696 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209699584 0 0) 0x7f82900b0a70 con 0x7f82900b4d60 2024-03-25T19:32:59.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:59.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:59.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:59.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:32:59.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 1 -- 172.21.15.123:0/1091694696 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82900b0d50 con 0x7f82900b4d60 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 1 -- 172.21.15.123:0/1091694696 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 733110881 0 0) 0x7f8278003520 con 0x7f82900b1730 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 1 -- 172.21.15.123:0/1091694696 >> v1:172.21.15.151:6789/0 conn(0x7f82900b4d60 legacy=0x7f829013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 1 -- 172.21.15.123:0/1091694696 >> v1:172.21.15.123:6789/0 conn(0x7f829014be90 legacy=0x7f829014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 1 -- 172.21.15.123:0/1091694696 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82900b1030 con 0x7f82900b1730 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 1 -- 172.21.15.123:0/1091694696 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8284001690 con 0x7f82900b1730 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:59.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f828e7fc640 10 monclient: dump: 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f8295cb3640 5 monclient: authenticate success, global_id 4700 2024-03-25T19:32:59.574 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.571+0000 7f8295cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:59.575 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.575+0000 7f828e7fc640 1 -- 172.21.15.123:0/1091694696 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82840017f0 con 0x7f82900b1730 2024-03-25T19:32:59.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.575+0000 7f828e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:59.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.575+0000 7f8295cb3640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:59.576 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.575+0000 7f8295cb3640 4 set_mon_vals no callback set 2024-03-25T19:32:59.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.576+0000 7f8295cb3640 10 monclient: shutdown 2024-03-25T19:32:59.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.576+0000 7f8295cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:59.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.576+0000 7f8295cb3640 1 -- 172.21.15.123:0/1091694696 >> v1:172.21.15.123:6790/0 conn(0x7f82900b1730 legacy=0x7f82900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.576+0000 7f8295cb3640 1 -- 172.21.15.123:0/1091694696 shutdown_connections 2024-03-25T19:32:59.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.576+0000 7f8295cb3640 1 -- 172.21.15.123:0/1091694696 >> 172.21.15.123:0/1091694696 conn(0x7f82900617a0 msgr2=0x7f8290061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:59.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.576+0000 7f8295cb3640 1 -- 172.21.15.123:0/1091694696 shutdown_connections 2024-03-25T19:32:59.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.576+0000 7f8295cb3640 1 -- 172.21.15.123:0/1091694696 wait complete. 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.576+0000 7f8295cb3640 10 monclient: build_initial_monmap 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.576+0000 7f8295cb3640 10 monclient: monmap: 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:32:59.579 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.577+0000 7f8295cb3640 1 librados: starting msgr at 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.577+0000 7f8295cb3640 1 librados: starting objecter 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.577+0000 7f8295cb3640 1 Processor -- start 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.577+0000 7f8295cb3640 1 -- start start 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.577+0000 7f8295cb3640 1 librados: setting wanted keys 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.577+0000 7f8295cb3640 1 librados: calling monclient init 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.577+0000 7f8295cb3640 10 monclient: init 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.578+0000 7f8295cb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.578+0000 7f8295cb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.578+0000 7f8295cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f82900b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.578+0000 7f8295cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f82900b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.578+0000 7f8295cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f829014be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.578+0000 7f8295cb3640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82901daf00 con 0x7f82900b1730 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.578+0000 7f8295cb3640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8290138820 con 0x7f82900b4d60 2024-03-25T19:32:59.580 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.578+0000 7f8295cb3640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82901dab40 con 0x7f829014be90 2024-03-25T19:32:59.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.578+0000 7f8295cb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:59.581 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.578+0000 7f8295cb3640 10 monclient(hunting): authenticate will time out at 6216.939026s 2024-03-25T19:32:59.585 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.585+0000 7f828effd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f82900b4d60 0x7f8290138110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:40698/0 (socket says 172.21.15.123:40698) 2024-03-25T19:32:59.585 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.585+0000 7f828effd640 1 -- 172.21.15.123:0/4195400571 learned_addr learned my addr 172.21.15.123:0/4195400571 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:32:59.586 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.586+0000 7f8277fff640 10 client.?.objecter ms_handle_connect 0x7f82900b4d60 2024-03-25T19:32:59.586 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.586+0000 7f8277fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:59.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.595+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559916905 0 0) 0x7f8290138820 con 0x7f82900b4d60 2024-03-25T19:32:59.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.595+0000 7f8277fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:59.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.595+0000 7f8277fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:59.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.595+0000 7f8277fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:59.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.595+0000 7f8277fff640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:32:59.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.595+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f827c004eb0 con 0x7f82900b4d60 2024-03-25T19:32:59.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.595+0000 7f8277fff640 10 client.?.objecter ms_handle_connect 0x7f82900b1730 2024-03-25T19:32:59.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.595+0000 7f8277fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:59.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.596+0000 7f8277fff640 10 client.?.objecter ms_handle_connect 0x7f829014be90 2024-03-25T19:32:59.597 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.596+0000 7f8277fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:59.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.610+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484677658 0 0) 0x7f82901daf00 con 0x7f82900b1730 2024-03-25T19:32:59.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.610+0000 7f8277fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:59.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.610+0000 7f8277fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:59.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.610+0000 7f8277fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:59.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.610+0000 7f8277fff640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:32:59.611 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.610+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8290138820 con 0x7f82900b1730 2024-03-25T19:32:59.612 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.611+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1351780347 0 0) 0x7f827c004eb0 con 0x7f82900b4d60 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.611+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82901daf00 con 0x7f82900b4d60 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.611+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8284001ce0 con 0x7f82900b4d60 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.611+0000 7f8277fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.611+0000 7f8277fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.611+0000 7f8277fff640 10 monclient(hunting): dump: 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:59.613 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:59.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.611+0000 7f8277fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:59.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.612+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385138676 0 0) 0x7f82901dab40 con 0x7f829014be90 2024-03-25T19:32:59.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.612+0000 7f8277fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:59.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.612+0000 7f8277fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:59.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.612+0000 7f8277fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:59.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.612+0000 7f8277fff640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:32:59.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.612+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f827c004eb0 con 0x7f829014be90 2024-03-25T19:32:59.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 460616980 0 0) 0x7f8290138820 con 0x7f82900b1730 2024-03-25T19:32:59.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82901dab40 con 0x7f82900b1730 2024-03-25T19:32:59.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f82800019f0 con 0x7f82900b1730 2024-03-25T19:32:59.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:59.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:59.631 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 10 monclient(hunting): dump: 2024-03-25T19:32:59.631 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3881155594 0 0) 0x7f82901daf00 con 0x7f82900b4d60 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 >> v1:172.21.15.123:6790/0 conn(0x7f829014be90 legacy=0x7f82901e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 >> v1:172.21.15.123:6789/0 conn(0x7f82900b1730 legacy=0x7f82901db420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 1 monclient: found mon.b 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:59.632 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82901dacd0 con 0x7f82900b4d60 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 10 monclient: _finish_auth 0 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8277fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8295cb3640 5 monclient: authenticate success, global_id 4690 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8295cb3640 10 monclient: _renew_subs 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8295cb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8295cb3640 1 -- 172.21.15.123:0/4195400571 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8290142bb0 con 0x7f82900b4d60 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8295cb3640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8295cb3640 10 monclient: _renew_subs 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8295cb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8295cb3640 1 -- 172.21.15.123:0/4195400571 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8290142940 con 0x7f82900b4d60 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.629+0000 7f8295cb3640 1 librados: init done 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.630+0000 7f8295cb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.630+0000 7f8295cb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:59.633 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.630+0000 7f8295cb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:32:59.634 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.630+0000 7f8295cb3640 1 -- 172.21.15.123:0/4195400571 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82900627b0 con 0x7f82900b4d60 2024-03-25T19:32:59.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.920+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8284004a90 con 0x7f82900b4d60 2024-03-25T19:32:59.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.920+0000 7f8277fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:59.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.920+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f8284004e60 con 0x7f82900b4d60 2024-03-25T19:32:59.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.920+0000 7f8277fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:59.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.920+0000 7f8277fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:59.921 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.920+0000 7f8277fff640 10 monclient: dump: 2024-03-25T19:32:59.921 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:32:59.921 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:32:59.922 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:59.922 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:32:59.922 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:59.922 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:32:59.922 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:32:59.922 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:32:59.922 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:32:59.922 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:32:59.922 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.920+0000 7f828dffb640 4 set_mon_vals no callback set 2024-03-25T19:32:59.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.921+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f8284005040 con 0x7f82900b4d60 2024-03-25T19:32:59.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.921+0000 7f828f7fe640 10 monclient: get_auth_request con 0x7f827c06dfb0 auth_method 0 2024-03-25T19:32:59.923 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.922+0000 7f8277fff640 10 client.4690.objecter ms_handle_connect 0x7f827c06dfb0 2024-03-25T19:32:59.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.926+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7f8284054a20 con 0x7f82900b4d60 2024-03-25T19:32:59.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.926+0000 7f8277fff640 10 client.4690.objecter ms_dispatch 0x7f8290000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:32:59.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.926+0000 7f8277fff640 3 client.4690.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:32:59.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.926+0000 7f8277fff640 3 client.4690.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:32:59.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.926+0000 7f8277fff640 20 client.4690.objecter dump_active .. 0 homeless 2024-03-25T19:32:59.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.927+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8284050530 con 0x7f82900b4d60 2024-03-25T19:32:59.927 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.927+0000 7f8277fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:59.928 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:32:59.927+0000 7f8277fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.184+0000 7f8295cb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.184+0000 7f8295cb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.184+0000 7f8295cb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.184+0000 7f8295cb3640 1 -- 172.21.15.123:0/4195400571 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f829014bc70 con 0x7f82900b4d60 2024-03-25T19:33:00.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.217+0000 7f8277fff640 1 -- 172.21.15.123:0/4195400571 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7f8284052460 con 0x7f82900b4d60 2024-03-25T19:33:00.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.217+0000 7f8277fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:00.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.217+0000 7f8277fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:00.219 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:33:00.219 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:00.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.219+0000 7f8295cb3640 10 librados: watch_flush enter 2024-03-25T19:33:00.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 10 librados: watch_flush exit 2024-03-25T19:33:00.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 20 client.4690.objecter shutdown clearing up homeless session... 2024-03-25T19:33:00.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 10 client.4690.objecter successfully canceled tick 2024-03-25T19:33:00.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 1 -- 172.21.15.123:0/4195400571 >> v1:172.21.15.151:6800/727684239 conn(0x7f827c06dfb0 legacy=0x7f827c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 10 monclient: shutdown 2024-03-25T19:33:00.221 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:00.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 1 -- 172.21.15.123:0/4195400571 >> v1:172.21.15.151:6789/0 conn(0x7f82900b4d60 legacy=0x7f8290138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 1 -- 172.21.15.123:0/4195400571 shutdown_connections 2024-03-25T19:33:00.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 1 -- 172.21.15.123:0/4195400571 >> 172.21.15.123:0/4195400571 conn(0x7f82900617a0 msgr2=0x7f82900b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:00.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 1 -- 172.21.15.123:0/4195400571 shutdown_connections 2024-03-25T19:33:00.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 1 -- 172.21.15.123:0/4195400571 wait complete. 2024-03-25T19:33:00.222 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:00.220+0000 7f8295cb3640 1 librados: shutdown 2024-03-25T19:33:00.234 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:07.041 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:07.144 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.145+0000 7fb1e2dc6640 10 monclient: get_monmap_and_config 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.145+0000 7fb1e2dc6640 10 monclient: build_initial_monmap 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.145+0000 7fb1e2dc6640 10 monclient: monmap: 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:33:07.146140+0000 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:33:07.146140+0000 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:33:07.145 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:07.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.146+0000 7fb1e2dc6640 1 Processor -- start 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.146+0000 7fb1e2dc6640 1 -- start start 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.146+0000 7fb1e2dc6640 10 monclient: init 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.146+0000 7fb1e2dc6640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.147+0000 7fb1e2dc6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.147+0000 7fb1e2dc6640 10 monclient(hunting): picked mon.noname-a con 0x7fb1dc0b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.147+0000 7fb1e2dc6640 10 monclient(hunting): picked mon.noname-c con 0x7fb1dc0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.147+0000 7fb1e2dc6640 10 monclient(hunting): picked mon.noname-b con 0x7fb1dc14be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.147+0000 7fb1e2dc6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc0af940 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.147+0000 7fb1e2dc6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc0b0a70 con 0x7fb1dc0b1730 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.147+0000 7fb1e2dc6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc0b0d50 con 0x7fb1dc14be90 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.147+0000 7fb1e2dc6640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:07.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.147+0000 7fb1e2dc6640 10 monclient(hunting): authenticate will time out at 6224.507015s 2024-03-25T19:33:07.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.160+0000 7fb1e1363640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fb1dc14be90 0x7fb1dc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:50618/0 (socket says 172.21.15.123:50618) 2024-03-25T19:33:07.159 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.160+0000 7fb1e1363640 1 -- 172.21.15.123:0/991419969 learned_addr learned my addr 172.21.15.123:0/991419969 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:07.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.178+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121190670 0 0) 0x7fb1dc0b0d50 con 0x7fb1dc14be90 2024-03-25T19:33:07.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.178+0000 7fb1db7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.178+0000 7fb1db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.178+0000 7fb1db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.178+0000 7fb1db7fe640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:33:07.178 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.178+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1cc003520 con 0x7fb1dc14be90 2024-03-25T19:33:07.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.186+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035656274 0 0) 0x7fb1dc0b0a70 con 0x7fb1dc0b1730 2024-03-25T19:33:07.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.186+0000 7fb1db7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.186+0000 7fb1db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.186+0000 7fb1db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.186+0000 7fb1db7fe640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:33:07.185 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.186+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1dc0b0d50 con 0x7fb1dc0b1730 2024-03-25T19:33:07.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.187+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927740800 0 0) 0x7fb1dc0af940 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.187+0000 7fb1db7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.187+0000 7fb1db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.187+0000 7fb1db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.187+0000 7fb1db7fe640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:33:07.187 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.187+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1dc0b0a70 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3265021674 0 0) 0x7fb1dc0b0d50 con 0x7fb1dc0b1730 2024-03-25T19:33:07.191 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 >> v1:172.21.15.123:6790/0 conn(0x7fb1dc14be90 legacy=0x7fb1dc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 >> v1:172.21.15.123:6789/0 conn(0x7fb1dc0b4d60 legacy=0x7fb1dc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 1 monclient: found mon.noname-c 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc0b1030 con 0x7fb1dc0b1730 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb1c8001670 con 0x7fb1dc0b1730 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1e2dc6640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1e2dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:07.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1db7fe640 10 monclient: dump: 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:07.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.191+0000 7fb1e2dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:07.208 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1c80017d0 con 0x7fb1dc0b1730 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1db7fe640 1 -- 172.21.15.123:0/991419969 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb1c8001be0 con 0x7fb1dc0b1730 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1e2dc6640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1e2dc6640 4 set_mon_vals no callback set 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1db7fe640 10 monclient: dump: 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:07.209 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:07.210 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:07.210 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:07.210 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:07.210 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1e2dc6640 10 monclient: shutdown 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1e2dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/991419969 >> v1:172.21.15.151:6789/0 conn(0x7fb1dc0b1730 legacy=0x7fb1dc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/991419969 shutdown_connections 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/991419969 >> 172.21.15.123:0/991419969 conn(0x7fb1dc0617a0 msgr2=0x7fb1dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.209+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/991419969 shutdown_connections 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.210+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/991419969 wait complete. 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.210+0000 7fb1e2dc6640 10 monclient: build_initial_monmap 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.210+0000 7fb1e2dc6640 10 monclient: monmap: 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:07.212 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.211+0000 7fb1e2dc6640 1 librados: starting msgr at 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.211+0000 7fb1e2dc6640 1 librados: starting objecter 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.211+0000 7fb1e2dc6640 1 Processor -- start 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.211+0000 7fb1e2dc6640 1 -- start start 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.211+0000 7fb1e2dc6640 1 librados: setting wanted keys 2024-03-25T19:33:07.213 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.211+0000 7fb1e2dc6640 1 librados: calling monclient init 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.211+0000 7fb1e2dc6640 10 monclient: init 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.212+0000 7fb1e2dc6640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.212+0000 7fb1e2dc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.212+0000 7fb1e2dc6640 10 monclient(hunting): picked mon.noname-b con 0x7fb1dc0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.212+0000 7fb1e2dc6640 10 monclient(hunting): picked mon.noname-a con 0x7fb1dc0b4d60 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.212+0000 7fb1e2dc6640 10 monclient(hunting): picked mon.noname-c con 0x7fb1dc14be90 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.212+0000 7fb1e2dc6640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc1d77e0 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.212+0000 7fb1e2dc6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc13ae20 con 0x7fb1dc0b1730 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.212+0000 7fb1e2dc6640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc1d8380 con 0x7fb1dc14be90 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.212+0000 7fb1e2dc6640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:07.214 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.212+0000 7fb1e2dc6640 10 monclient(hunting): authenticate will time out at 6224.572207s 2024-03-25T19:33:07.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.229+0000 7fb1e0b62640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fb1dc0b1730 0x7fb1dc1d7c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:37238/0 (socket says 172.21.15.123:37238) 2024-03-25T19:33:07.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.229+0000 7fb1e0b62640 1 -- 172.21.15.123:0/609811629 learned_addr learned my addr 172.21.15.123:0/609811629 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:07.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.230+0000 7fb1d97fa640 10 client.?.objecter ms_handle_connect 0x7fb1dc0b1730 2024-03-25T19:33:07.230 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.230+0000 7fb1d97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:07.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.234+0000 7fb1d97fa640 10 client.?.objecter ms_handle_connect 0x7fb1dc0b4d60 2024-03-25T19:33:07.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.234+0000 7fb1d97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:07.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.235+0000 7fb1d97fa640 10 client.?.objecter ms_handle_connect 0x7fb1dc14be90 2024-03-25T19:33:07.234 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.235+0000 7fb1d97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:07.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.243+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360859459 0 0) 0x7fb1dc1d77e0 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.243+0000 7fb1d97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.243+0000 7fb1d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.243+0000 7fb1d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.243+0000 7fb1d97fa640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:33:07.243 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.243+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c0004eb0 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.246+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432121950 0 0) 0x7fb1dc13ae20 con 0x7fb1dc0b1730 2024-03-25T19:33:07.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.246+0000 7fb1d97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.246+0000 7fb1d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.246+0000 7fb1d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.246+0000 7fb1d97fa640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:33:07.246 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.246+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1dc1d77e0 con 0x7fb1dc0b1730 2024-03-25T19:33:07.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.252+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931009951 0 0) 0x7fb1dc1d8380 con 0x7fb1dc14be90 2024-03-25T19:33:07.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.252+0000 7fb1d97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.252+0000 7fb1d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.252+0000 7fb1d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.252+0000 7fb1d97fa640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:33:07.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.252+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1dc13ae20 con 0x7fb1dc14be90 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.259+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3577890132 0 0) 0x7fb1c0004eb0 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.259+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1dc1d8380 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.259+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb1c8001ba0 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.259+0000 7fb1d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.259+0000 7fb1d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.259+0000 7fb1d97fa640 10 monclient(hunting): dump: 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:07.259 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:07.260 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:07.260 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:07.260 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:07.260 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:07.260 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:07.260 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.259+0000 7fb1d97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.260+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3386833446 0 0) 0x7fb1dc1d77e0 con 0x7fb1dc0b1730 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.260+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1c0004eb0 con 0x7fb1dc0b1730 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.260+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb1d00019f0 con 0x7fb1dc0b1730 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.260+0000 7fb1d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.260+0000 7fb1d97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.260+0000 7fb1d97fa640 10 monclient(hunting): dump: 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:07.261 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:07.262 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:07.262 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:07.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2595951446 0 0) 0x7fb1dc1d8380 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 >> v1:172.21.15.123:6790/0 conn(0x7fb1dc14be90 legacy=0x7fb1dc1e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 >> v1:172.21.15.151:6789/0 conn(0x7fb1dc0b1730 legacy=0x7fb1dc1d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:33:07.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 1 monclient: found mon.a 2024-03-25T19:33:07.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:07.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:07.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc139b20 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 10 monclient: _finish_auth 0 2024-03-25T19:33:07.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:33:07.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:07.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1e2dc6640 5 monclient: authenticate success, global_id 4686 2024-03-25T19:33:07.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1e2dc6640 10 monclient: _renew_subs 2024-03-25T19:33:07.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1e2dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:07.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/609811629 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc143030 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1e2dc6640 10 client.4686.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:07.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1e2dc6640 10 monclient: _renew_subs 2024-03-25T19:33:07.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1e2dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:07.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/609811629 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc13b1d0 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.273+0000 7fb1e2dc6640 1 librados: init done 2024-03-25T19:33:07.275 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.274+0000 7fb1e2dc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:07.275 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.274+0000 7fb1e2dc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:07.275 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.274+0000 7fb1e2dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:07.275 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.274+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/609811629 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc0627b0 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.602+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1c80038a0 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.602+0000 7fb1d97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:07.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.602+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fb1c8003bb0 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.602+0000 7fb1d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:07.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.602+0000 7fb1d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:07.602 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.602+0000 7fb1d97fa640 10 monclient: dump: 2024-03-25T19:33:07.602 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:07.602 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:07.602 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:07.603 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:07.603 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:07.603 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:07.603 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:07.603 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:07.603 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:07.603 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:07.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.602+0000 7fb1daffd640 4 set_mon_vals no callback set 2024-03-25T19:33:07.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.603+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fb1c8019b80 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.603+0000 7fb1e0b62640 10 monclient: get_auth_request con 0x7fb1c006df70 auth_method 0 2024-03-25T19:33:07.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.604+0000 7fb1d97fa640 10 client.4686.objecter ms_handle_connect 0x7fb1c006df70 2024-03-25T19:33:07.609 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.609+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7fb1c8053750 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.609 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.609+0000 7fb1d97fa640 10 client.4686.objecter ms_dispatch 0x7fb1dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:33:07.609 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.609+0000 7fb1d97fa640 3 client.4686.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:33:07.609 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.609+0000 7fb1d97fa640 3 client.4686.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:33:07.609 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.609+0000 7fb1d97fa640 20 client.4686.objecter dump_active .. 0 homeless 2024-03-25T19:33:07.609 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.609+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1c804f260 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.609 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.609+0000 7fb1d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:07.609 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.609+0000 7fb1d97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:07.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.866+0000 7fb1e2dc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:07.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.866+0000 7fb1e2dc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:07.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.866+0000 7fb1e2dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:07.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.866+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/609811629 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1dc1d7990 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.890+0000 7fb1d97fa640 1 -- 172.21.15.123:0/609811629 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7fb1dc1d7990 con 0x7fb1dc0b4d60 2024-03-25T19:33:07.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.890+0000 7fb1d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:07.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.890+0000 7fb1d97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:07.891 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:33:07.891 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.892+0000 7fb1e2dc6640 10 librados: watch_flush enter 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 10 librados: watch_flush exit 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 20 client.4686.objecter shutdown clearing up homeless session... 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 10 client.4686.objecter successfully canceled tick 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/609811629 >> v1:172.21.15.151:6800/727684239 conn(0x7fb1c006df70 legacy=0x7fb1c0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 10 monclient: shutdown 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/609811629 >> v1:172.21.15.123:6789/0 conn(0x7fb1dc0b4d60 legacy=0x7fb1dc139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/609811629 shutdown_connections 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/609811629 >> 172.21.15.123:0/609811629 conn(0x7fb1dc0617a0 msgr2=0x7fb1dc13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/609811629 shutdown_connections 2024-03-25T19:33:07.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 1 -- 172.21.15.123:0/609811629 wait complete. 2024-03-25T19:33:07.894 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:07.893+0000 7fb1e2dc6640 1 librados: shutdown 2024-03-25T19:33:07.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:14.712 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.825+0000 7f86dc9e8640 10 monclient: get_monmap_and_config 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.825+0000 7f86dc9e8640 10 monclient: build_initial_monmap 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.825+0000 7f86dc9e8640 10 monclient: monmap: 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:33:14.826140+0000 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:33:14.826140+0000 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:14.825 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:33:14.826 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:33:14.826 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:14.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.825+0000 7f86dc9e8640 1 Processor -- start 2024-03-25T19:33:14.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.826+0000 7f86dc9e8640 1 -- start start 2024-03-25T19:33:14.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.826+0000 7f86dc9e8640 10 monclient: init 2024-03-25T19:33:14.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.826+0000 7f86dc9e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:14.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.826+0000 7f86dc9e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:14.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.826+0000 7f86dc9e8640 10 monclient(hunting): picked mon.noname-b con 0x7f86d80a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:14.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.827+0000 7f86dc9e8640 10 monclient(hunting): picked mon.noname-c con 0x7f86d813f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:14.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.827+0000 7f86dc9e8640 10 monclient(hunting): picked mon.noname-a con 0x7f86d81430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:14.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.827+0000 7f86dc9e8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d8142520 con 0x7f86d81430c0 2024-03-25T19:33:14.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.827+0000 7f86dc9e8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d80b28a0 con 0x7f86d813f710 2024-03-25T19:33:14.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.827+0000 7f86dc9e8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d80b2a30 con 0x7f86d80a98f0 2024-03-25T19:33:14.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.827+0000 7f86dc9e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:14.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.827+0000 7f86dc9e8640 10 monclient(hunting): authenticate will time out at 6232.187060s 2024-03-25T19:33:14.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.830+0000 7f86d5d9b640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f86d813f710 0x7f86d81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:42034/0 (socket says 172.21.15.123:42034) 2024-03-25T19:33:14.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.830+0000 7f86d5d9b640 1 -- 172.21.15.123:0/2383527360 learned_addr learned my addr 172.21.15.123:0/2383527360 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:14.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.836+0000 7f86d559a640 1 -- 172.21.15.123:0/2383527360 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737182576 0 0) 0x7f86d80b28a0 con 0x7f86d813f710 2024-03-25T19:33:14.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.836+0000 7f86d559a640 10 monclient(hunting): _negotiate 2024-03-25T19:33:14.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.836+0000 7f86d559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:14.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.836+0000 7f86d559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:14.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.836+0000 7f86d559a640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:33:14.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.836+0000 7f86d559a640 1 -- 172.21.15.123:0/2383527360 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c0003520 con 0x7f86d813f710 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 1 -- 172.21.15.123:0/2383527360 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3981137248 0 0) 0x7f86c0003520 con 0x7f86d813f710 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 1 -- 172.21.15.123:0/2383527360 >> v1:172.21.15.123:6790/0 conn(0x7f86d80a98f0 legacy=0x7f86d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 1 -- 172.21.15.123:0/2383527360 >> v1:172.21.15.123:6789/0 conn(0x7f86d81430c0 legacy=0x7f86d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 10 monclient: _finish_hunting 0 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 1 monclient: found mon.noname-c 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 1 -- 172.21.15.123:0/2383527360 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d80b44c0 con 0x7f86d813f710 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 10 monclient: _finish_auth 0 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 10 monclient: _check_auth_tickets 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 1 -- 172.21.15.123:0/2383527360 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f86cc001690 con 0x7f86d813f710 2024-03-25T19:33:14.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86d559a640 10 monclient: dump: 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86dc9e8640 5 monclient: authenticate success, global_id 4699 2024-03-25T19:33:14.844 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.842+0000 7f86dc9e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:14.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.858+0000 7f86d559a640 1 -- 172.21.15.123:0/2383527360 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86cc001860 con 0x7f86d813f710 2024-03-25T19:33:14.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.858+0000 7f86d559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:14.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.858+0000 7f86d559a640 1 -- 172.21.15.123:0/2383527360 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f86cc001cb0 con 0x7f86d813f710 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.858+0000 7f86dc9e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.858+0000 7f86dc9e8640 4 set_mon_vals no callback set 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.859+0000 7f86d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.859+0000 7f86d559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.859+0000 7f86d559a640 10 monclient: dump: 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:14.859 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:14.860 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:14.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.859+0000 7f86dc9e8640 10 monclient: shutdown 2024-03-25T19:33:14.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.859+0000 7f86dc9e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:14.861 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.859+0000 7f86dc9e8640 1 -- 172.21.15.123:0/2383527360 >> v1:172.21.15.151:6789/0 conn(0x7f86d813f710 legacy=0x7f86d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.860+0000 7f86dc9e8640 1 -- 172.21.15.123:0/2383527360 shutdown_connections 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.860+0000 7f86dc9e8640 1 -- 172.21.15.123:0/2383527360 >> 172.21.15.123:0/2383527360 conn(0x7f86d80b7ea0 msgr2=0x7f86d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.860+0000 7f86dc9e8640 1 -- 172.21.15.123:0/2383527360 shutdown_connections 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.860+0000 7f86dc9e8640 1 -- 172.21.15.123:0/2383527360 wait complete. 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.861+0000 7f86dc9e8640 10 monclient: build_initial_monmap 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.861+0000 7f86dc9e8640 10 monclient: monmap: 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:33:14.862 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.861+0000 7f86dc9e8640 1 librados: starting msgr at 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.861+0000 7f86dc9e8640 1 librados: starting objecter 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.861+0000 7f86dc9e8640 1 Processor -- start 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.861+0000 7f86dc9e8640 1 -- start start 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.861+0000 7f86dc9e8640 1 librados: setting wanted keys 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.861+0000 7f86dc9e8640 1 librados: calling monclient init 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.861+0000 7f86dc9e8640 10 monclient: init 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.862+0000 7f86dc9e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.862+0000 7f86dc9e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.862+0000 7f86dc9e8640 10 monclient(hunting): picked mon.noname-b con 0x7f86d80a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.862+0000 7f86dc9e8640 10 monclient(hunting): picked mon.noname-c con 0x7f86d813f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:14.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.862+0000 7f86dc9e8640 10 monclient(hunting): picked mon.noname-a con 0x7f86d81430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:14.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.862+0000 7f86dc9e8640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d81d7780 con 0x7f86d81430c0 2024-03-25T19:33:14.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.862+0000 7f86dc9e8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d8137990 con 0x7f86d80a98f0 2024-03-25T19:33:14.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.862+0000 7f86dc9e8640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d81d7380 con 0x7f86d813f710 2024-03-25T19:33:14.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.862+0000 7f86dc9e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:14.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.862+0000 7f86dc9e8640 10 monclient(hunting): authenticate will time out at 6232.222032s 2024-03-25T19:33:14.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.865+0000 7f86d5d9b640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f86d813f710 0x7f86d8138170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:55500/0 (socket says 172.21.15.123:55500) 2024-03-25T19:33:14.864 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.865+0000 7f86d5d9b640 1 -- 172.21.15.123:0/1173945539 learned_addr learned my addr 172.21.15.123:0/1173945539 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:14.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.867+0000 7f86c6ffd640 10 client.?.objecter ms_handle_connect 0x7f86d813f710 2024-03-25T19:33:14.866 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.867+0000 7f86c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:14.872 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.872+0000 7f86c6ffd640 10 client.?.objecter ms_handle_connect 0x7f86d81430c0 2024-03-25T19:33:14.872 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.872+0000 7f86c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:14.873 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.873+0000 7f86c6ffd640 10 client.?.objecter ms_handle_connect 0x7f86d80a98f0 2024-03-25T19:33:14.873 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.873+0000 7f86c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:14.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.880+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940282469 0 0) 0x7f86d8137990 con 0x7f86d80a98f0 2024-03-25T19:33:14.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.880+0000 7f86c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:14.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.880+0000 7f86c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:14.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.880+0000 7f86c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:14.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.880+0000 7f86c6ffd640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:33:14.880 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.880+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b8004eb0 con 0x7f86d80a98f0 2024-03-25T19:33:14.884 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.884+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174833020 0 0) 0x7f86d81d7380 con 0x7f86d813f710 2024-03-25T19:33:14.884 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.884+0000 7f86c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:14.884 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.884+0000 7f86c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:14.884 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.884+0000 7f86c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:14.884 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.884+0000 7f86c6ffd640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:33:14.884 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.884+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d8137990 con 0x7f86d813f710 2024-03-25T19:33:14.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.895+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626661045 0 0) 0x7f86d81d7780 con 0x7f86d81430c0 2024-03-25T19:33:14.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.895+0000 7f86c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:14.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.895+0000 7f86c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:14.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.895+0000 7f86c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:14.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.895+0000 7f86c6ffd640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:33:14.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.895+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d81d7380 con 0x7f86d81430c0 2024-03-25T19:33:14.910 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.910+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1396115191 0 0) 0x7f86b8004eb0 con 0x7f86d80a98f0 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.910+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d81d7780 con 0x7f86d80a98f0 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.910+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f86c8001930 con 0x7f86d80a98f0 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.910+0000 7f86c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.910+0000 7f86c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.910+0000 7f86c6ffd640 10 monclient(hunting): dump: 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:14.911 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:14.912 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:14.912 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:14.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.911+0000 7f86c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:14.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.912+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2926199182 0 0) 0x7f86d8137990 con 0x7f86d813f710 2024-03-25T19:33:14.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.912+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86b8004eb0 con 0x7f86d813f710 2024-03-25T19:33:14.912 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.912+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f86cc001b70 con 0x7f86d813f710 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.912+0000 7f86c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.912+0000 7f86c6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.912+0000 7f86c6ffd640 10 monclient(hunting): dump: 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:14.913 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:14.917 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.918+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2966159113 0 0) 0x7f86d81d7380 con 0x7f86d81430c0 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.918+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d8137990 con 0x7f86d81430c0 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.918+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f86d0001a10 con 0x7f86d81430c0 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.918+0000 7f86c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.918+0000 7f86c6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.918+0000 7f86c6ffd640 10 monclient(hunting): dump: 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:14.918 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:14.919 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:14.919 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:14.919 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:14.919 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:14.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4249975517 0 0) 0x7f86d8137990 con 0x7f86d81430c0 2024-03-25T19:33:14.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 >> v1:172.21.15.123:6790/0 conn(0x7f86d813f710 legacy=0x7f86d8138170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:14.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 >> v1:172.21.15.151:6789/0 conn(0x7f86d80a98f0 legacy=0x7f86d81d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:14.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:14.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 1 monclient: found mon.a 2024-03-25T19:33:14.948 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d8138d60 con 0x7f86d81430c0 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86dc9e8640 5 monclient: authenticate success, global_id 4689 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86dc9e8640 10 monclient: _renew_subs 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86dc9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86dc9e8640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d8138880 con 0x7f86d81430c0 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86dc9e8640 10 client.4689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86dc9e8640 10 monclient: _renew_subs 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86dc9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86dc9e8640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d81e34f0 con 0x7f86d81430c0 2024-03-25T19:33:14.949 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.948+0000 7f86dc9e8640 1 librados: init done 2024-03-25T19:33:14.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.949+0000 7f86dc9e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:14.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.949+0000 7f86dc9e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:14.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.949+0000 7f86dc9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:14.950 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:14.949+0000 7f86dc9e8640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d81d7510 con 0x7f86d81430c0 2024-03-25T19:33:15.257 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.257+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86d0001f20 con 0x7f86d81430c0 2024-03-25T19:33:15.257 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.257+0000 7f86c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.257+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f86d0003b10 con 0x7f86d81430c0 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.257+0000 7f86c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.257+0000 7f86c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.257+0000 7f86c6ffd640 10 monclient: dump: 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:15.258 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:15.259 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:15.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.257+0000 7f86d4d99640 4 set_mon_vals no callback set 2024-03-25T19:33:15.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.259+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f86d0019ae0 con 0x7f86d81430c0 2024-03-25T19:33:15.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.259+0000 7f86d659c640 10 monclient: get_auth_request con 0x7f86b806e110 auth_method 0 2024-03-25T19:33:15.259 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.259+0000 7f86c6ffd640 10 client.4689.objecter ms_handle_connect 0x7f86b806e110 2024-03-25T19:33:15.265 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.266+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7f86d0053730 con 0x7f86d81430c0 2024-03-25T19:33:15.265 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.266+0000 7f86c6ffd640 10 client.4689.objecter ms_dispatch 0x7f86d8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:33:15.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.266+0000 7f86c6ffd640 3 client.4689.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:33:15.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.266+0000 7f86c6ffd640 3 client.4689.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:33:15.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.266+0000 7f86c6ffd640 20 client.4689.objecter dump_active .. 0 homeless 2024-03-25T19:33:15.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.266+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86d004f240 con 0x7f86d81430c0 2024-03-25T19:33:15.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.266+0000 7f86c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:15.266 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.266+0000 7f86c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:15.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.510+0000 7f86dc9e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:15.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.510+0000 7f86dc9e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:15.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.510+0000 7f86dc9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:15.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.510+0000 7f86dc9e8640 1 -- 172.21.15.123:0/1173945539 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86d81d7930 con 0x7f86d81430c0 2024-03-25T19:33:15.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.549+0000 7f86c6ffd640 1 -- 172.21.15.123:0/1173945539 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7f86d81d7930 con 0x7f86d81430c0 2024-03-25T19:33:15.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.549+0000 7f86c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:15.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.549+0000 7f86c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:15.550 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:33:15.550 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:15.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.552+0000 7f86dc9e8640 10 librados: watch_flush enter 2024-03-25T19:33:15.552 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.552+0000 7f86dc9e8640 10 librados: watch_flush exit 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.552+0000 7f86dc9e8640 20 client.4689.objecter shutdown clearing up homeless session... 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.552+0000 7f86dc9e8640 10 client.4689.objecter successfully canceled tick 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.552+0000 7f86dc9e8640 1 -- 172.21.15.123:0/1173945539 >> v1:172.21.15.151:6800/727684239 conn(0x7f86b806e110 legacy=0x7f86b80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.552+0000 7f86dc9e8640 10 monclient: shutdown 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.552+0000 7f86dc9e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.552+0000 7f86dc9e8640 1 -- 172.21.15.123:0/1173945539 >> v1:172.21.15.123:6789/0 conn(0x7f86d81430c0 legacy=0x7f86d81e1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.552+0000 7f86dc9e8640 1 -- 172.21.15.123:0/1173945539 shutdown_connections 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.552+0000 7f86dc9e8640 1 -- 172.21.15.123:0/1173945539 >> 172.21.15.123:0/1173945539 conn(0x7f86d80b7ea0 msgr2=0x7f86d813d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.553+0000 7f86dc9e8640 1 -- 172.21.15.123:0/1173945539 shutdown_connections 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.553+0000 7f86dc9e8640 1 -- 172.21.15.123:0/1173945539 wait complete. 2024-03-25T19:33:15.553 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:15.553+0000 7f86dc9e8640 1 librados: shutdown 2024-03-25T19:33:15.564 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:16.818 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:06.390 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:16.824 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:06.396 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:16.859 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:06.431 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:16.882 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:06.415 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:16.886 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:06.458 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:16.890 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:06.462 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:16.900 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:06.434 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:16.917 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:06.489 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:16.922 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:06.455 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:22.371 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:22.486 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.487+0000 7f18e5125640 10 monclient: get_monmap_and_config 2024-03-25T19:33:22.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.487+0000 7f18e5125640 10 monclient: build_initial_monmap 2024-03-25T19:33:22.487 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.487+0000 7f18e5125640 10 monclient: monmap: 2024-03-25T19:33:22.487 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:22.487 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:22.487 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:33:22.488090+0000 2024-03-25T19:33:22.487 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:33:22.488090+0000 2024-03-25T19:33:22.487 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:22.488 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:22.488 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:22.488 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:33:22.488 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:33:22.488 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.487+0000 7f18e5125640 1 Processor -- start 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.487+0000 7f18e5125640 1 -- start start 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.487+0000 7f18e5125640 10 monclient: init 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.488+0000 7f18e5125640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.488+0000 7f18e5125640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.488+0000 7f18e5125640 10 monclient(hunting): picked mon.noname-b con 0x7f18e00a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.488+0000 7f18e5125640 10 monclient(hunting): picked mon.noname-c con 0x7f18e013f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.488+0000 7f18e5125640 10 monclient(hunting): picked mon.noname-a con 0x7f18e01430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.488+0000 7f18e5125640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e0142520 con 0x7f18e01430c0 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.488+0000 7f18e5125640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00b28a0 con 0x7f18e013f710 2024-03-25T19:33:22.489 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.488+0000 7f18e5125640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00b2a30 con 0x7f18e00a98f0 2024-03-25T19:33:22.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.488+0000 7f18e5125640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:22.490 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.488+0000 7f18e5125640 10 monclient(hunting): authenticate will time out at 6239.848649s 2024-03-25T19:33:22.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.504+0000 7f18df59e640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f18e01430c0 0x7f18e014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:36494/0 (socket says 172.21.15.123:36494) 2024-03-25T19:33:22.503 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.504+0000 7f18df59e640 1 -- 172.21.15.123:0/973481327 learned_addr learned my addr 172.21.15.123:0/973481327 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:22.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.522+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296079143 0 0) 0x7f18e00b28a0 con 0x7f18e013f710 2024-03-25T19:33:22.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.522+0000 7f18ddd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.522+0000 7f18ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.522+0000 7f18ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.522+0000 7f18ddd9b640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:33:22.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.522+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c8003520 con 0x7f18e013f710 2024-03-25T19:33:22.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.526+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599887779 0 0) 0x7f18e0142520 con 0x7f18e01430c0 2024-03-25T19:33:22.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.526+0000 7f18ddd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.526+0000 7f18ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.526+0000 7f18ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.526+0000 7f18ddd9b640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:33:22.526 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.526+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e00b28a0 con 0x7f18e01430c0 2024-03-25T19:33:22.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.531+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328670719 0 0) 0x7f18e00b2a30 con 0x7f18e00a98f0 2024-03-25T19:33:22.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.531+0000 7f18ddd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.531+0000 7f18ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.531+0000 7f18ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.531+0000 7f18ddd9b640 10 monclient(hunting): my global_id is 4718 2024-03-25T19:33:22.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.531+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e0142520 con 0x7f18e00a98f0 2024-03-25T19:33:22.536 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3261406937 0 0) 0x7f18e00b28a0 con 0x7f18e01430c0 2024-03-25T19:33:22.536 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 >> v1:172.21.15.123:6790/0 conn(0x7f18e00a98f0 legacy=0x7f18e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.536 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 >> v1:172.21.15.151:6789/0 conn(0x7f18e013f710 legacy=0x7f18e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.536 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 1 monclient: found mon.noname-a 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e00b44c0 con 0x7f18e01430c0 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 10 monclient: _finish_auth 0 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f18d80019f0 con 0x7f18e01430c0 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18ddd9b640 10 monclient: dump: 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:22.537 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:22.538 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:22.538 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:22.538 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:22.538 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:22.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18e5125640 5 monclient: authenticate success, global_id 4692 2024-03-25T19:33:22.538 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.536+0000 7f18e5125640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:22.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.557+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d8001b50 con 0x7f18e01430c0 2024-03-25T19:33:22.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.557+0000 7f18ddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:22.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.557+0000 7f18ddd9b640 1 -- 172.21.15.123:0/973481327 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f18d8001f60 con 0x7f18e01430c0 2024-03-25T19:33:22.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.557+0000 7f18ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:22.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.557+0000 7f18ddd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:22.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.557+0000 7f18ddd9b640 10 monclient: dump: 2024-03-25T19:33:22.557 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.557+0000 7f18e5125640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:22.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.557+0000 7f18e5125640 4 set_mon_vals no callback set 2024-03-25T19:33:22.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.558+0000 7f18e5125640 10 monclient: shutdown 2024-03-25T19:33:22.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.558+0000 7f18e5125640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:22.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.558+0000 7f18e5125640 1 -- 172.21.15.123:0/973481327 >> v1:172.21.15.123:6789/0 conn(0x7f18e01430c0 legacy=0x7f18e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.558+0000 7f18e5125640 1 -- 172.21.15.123:0/973481327 shutdown_connections 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.558+0000 7f18e5125640 1 -- 172.21.15.123:0/973481327 >> 172.21.15.123:0/973481327 conn(0x7f18e00b7ea0 msgr2=0x7f18e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.558+0000 7f18e5125640 1 -- 172.21.15.123:0/973481327 shutdown_connections 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.558+0000 7f18e5125640 1 -- 172.21.15.123:0/973481327 wait complete. 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.559+0000 7f18e5125640 10 monclient: build_initial_monmap 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.560+0000 7f18e5125640 10 monclient: monmap: 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:33:22.560 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:22.561 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.560+0000 7f18e5125640 1 librados: starting msgr at 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.560+0000 7f18e5125640 1 librados: starting objecter 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 1 Processor -- start 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 1 -- start start 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 1 librados: setting wanted keys 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 1 librados: calling monclient init 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 10 monclient: init 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 10 monclient(hunting): picked mon.noname-c con 0x7f18e00a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 10 monclient(hunting): picked mon.noname-b con 0x7f18e013f710 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 10 monclient(hunting): picked mon.noname-a con 0x7f18e01430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e0135bf0 con 0x7f18e01430c0 2024-03-25T19:33:22.562 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00b6d10 con 0x7f18e013f710 2024-03-25T19:33:22.563 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e01317f0 con 0x7f18e00a98f0 2024-03-25T19:33:22.563 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:22.563 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.561+0000 7f18e5125640 10 monclient(hunting): authenticate will time out at 6239.921839s 2024-03-25T19:33:22.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.573+0000 7f18ded9d640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f18e00a98f0 0x7f18e01310e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:49066/0 (socket says 172.21.15.123:49066) 2024-03-25T19:33:22.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.573+0000 7f18ded9d640 1 -- 172.21.15.123:0/4210478612 learned_addr learned my addr 172.21.15.123:0/4210478612 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:22.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.578+0000 7f18c77fe640 10 client.?.objecter ms_handle_connect 0x7f18e013f710 2024-03-25T19:33:22.577 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.578+0000 7f18c77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:22.578 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.578+0000 7f18c77fe640 10 client.?.objecter ms_handle_connect 0x7f18e00a98f0 2024-03-25T19:33:22.578 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.578+0000 7f18c77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:22.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.582+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054128203 0 0) 0x7f18e00b6d10 con 0x7f18e013f710 2024-03-25T19:33:22.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.582+0000 7f18c77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.582+0000 7f18c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.582+0000 7f18c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.582+0000 7f18c77fe640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:33:22.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.582+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18cc004eb0 con 0x7f18e013f710 2024-03-25T19:33:22.585 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.586+0000 7f18c77fe640 10 client.?.objecter ms_handle_connect 0x7f18e01430c0 2024-03-25T19:33:22.585 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.586+0000 7f18c77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:22.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2757347107 0 0) 0x7f18cc004eb0 con 0x7f18e013f710 2024-03-25T19:33:22.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18e00b6d10 con 0x7f18e013f710 2024-03-25T19:33:22.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f18d4001950 con 0x7f18e013f710 2024-03-25T19:33:22.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:22.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:22.594 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 10 monclient(hunting): dump: 2024-03-25T19:33:22.594 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:22.594 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:22.594 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396069494 0 0) 0x7f18e01317f0 con 0x7f18e00a98f0 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:33:22.595 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.594+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18cc004eb0 con 0x7f18e00a98f0 2024-03-25T19:33:22.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.596+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681241012 0 0) 0x7f18e0135bf0 con 0x7f18e01430c0 2024-03-25T19:33:22.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.596+0000 7f18c77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.596+0000 7f18c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.596+0000 7f18c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.596+0000 7f18c77fe640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:33:22.596 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.596+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e01317f0 con 0x7f18e01430c0 2024-03-25T19:33:22.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 273981602 0 0) 0x7f18e00b6d10 con 0x7f18e013f710 2024-03-25T19:33:22.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 >> v1:172.21.15.123:6790/0 conn(0x7f18e00a98f0 legacy=0x7f18e01310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 >> v1:172.21.15.123:6789/0 conn(0x7f18e01430c0 legacy=0x7f18e01db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:22.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 1 monclient: found mon.b 2024-03-25T19:33:22.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:22.603 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0131980 con 0x7f18e013f710 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18e5125640 5 monclient: authenticate success, global_id 4708 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18e5125640 10 monclient: _renew_subs 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18e5125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18e5125640 1 -- 172.21.15.123:0/4210478612 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e00b76e0 con 0x7f18e013f710 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18e5125640 10 client.4708.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18e5125640 10 monclient: _renew_subs 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18e5125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18e5125640 1 -- 172.21.15.123:0/4210478612 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e0135270 con 0x7f18e013f710 2024-03-25T19:33:22.604 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.603+0000 7f18e5125640 1 librados: init done 2024-03-25T19:33:22.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.604+0000 7f18e5125640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:22.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.604+0000 7f18e5125640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:22.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.604+0000 7f18e5125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:22.605 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.604+0000 7f18e5125640 1 -- 172.21.15.123:0/4210478612 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e00627b0 con 0x7f18e013f710 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.874+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d4003610 con 0x7f18e013f710 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.874+0000 7f18c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.874+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f18d40039a0 con 0x7f18e013f710 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.874+0000 7f18c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.874+0000 7f18c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.874+0000 7f18c77fe640 10 monclient: dump: 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:22.874 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:22.875 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:22.875 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:22.875 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:22.875 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:22.875 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.874+0000 7f18dd59a640 4 set_mon_vals no callback set 2024-03-25T19:33:22.875 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.875+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f18d4003bc0 con 0x7f18e013f710 2024-03-25T19:33:22.875 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.875+0000 7f18ded9d640 10 monclient: get_auth_request con 0x7f18cc06de80 auth_method 0 2024-03-25T19:33:22.876 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.876+0000 7f18c77fe640 10 client.4708.objecter ms_handle_connect 0x7f18cc06de80 2024-03-25T19:33:22.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.882+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7f18d4053550 con 0x7f18e013f710 2024-03-25T19:33:22.882 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.882+0000 7f18c77fe640 10 client.4708.objecter ms_dispatch 0x7f18e0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:33:22.882 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.882+0000 7f18c77fe640 3 client.4708.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:33:22.882 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.882+0000 7f18c77fe640 3 client.4708.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:33:22.882 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.882+0000 7f18c77fe640 20 client.4708.objecter dump_active .. 0 homeless 2024-03-25T19:33:22.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.883+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18d404f060 con 0x7f18e013f710 2024-03-25T19:33:22.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.883+0000 7f18c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:22.883 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:22.883+0000 7f18c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.143+0000 7f18e5125640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.143+0000 7f18e5125640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.143+0000 7f18e5125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.143+0000 7f18e5125640 1 -- 172.21.15.123:0/4210478612 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18e013e100 con 0x7f18e013f710 2024-03-25T19:33:23.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.162+0000 7f18c77fe640 1 -- 172.21.15.123:0/4210478612 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7f18d4050f90 con 0x7f18e013f710 2024-03-25T19:33:23.161 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.162+0000 7f18c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:23.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.162+0000 7f18c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:23.162 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:33:23.162 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:23.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 10 librados: watch_flush enter 2024-03-25T19:33:23.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 10 librados: watch_flush exit 2024-03-25T19:33:23.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 20 client.4708.objecter shutdown clearing up homeless session... 2024-03-25T19:33:23.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 10 client.4708.objecter successfully canceled tick 2024-03-25T19:33:23.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 1 -- 172.21.15.123:0/4210478612 >> v1:172.21.15.151:6800/727684239 conn(0x7f18cc06de80 legacy=0x7f18cc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.164 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 10 monclient: shutdown 2024-03-25T19:33:23.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:23.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 1 -- 172.21.15.123:0/4210478612 >> v1:172.21.15.151:6789/0 conn(0x7f18e013f710 legacy=0x7f18e01dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 1 -- 172.21.15.123:0/4210478612 shutdown_connections 2024-03-25T19:33:23.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 1 -- 172.21.15.123:0/4210478612 >> 172.21.15.123:0/4210478612 conn(0x7f18e00b7ea0 msgr2=0x7f18e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:23.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 1 -- 172.21.15.123:0/4210478612 shutdown_connections 2024-03-25T19:33:23.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 1 -- 172.21.15.123:0/4210478612 wait complete. 2024-03-25T19:33:23.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:23.164+0000 7f18e5125640 1 librados: shutdown 2024-03-25T19:33:23.176 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:28.014 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:17.586 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:28.020 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:17.593 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:28.024 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:17.597 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:28.027 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:17.600 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:28.028 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:17.562 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:28.031 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:17.565 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:28.042 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:17.615 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:28.050 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:17.584 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:28.109 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:17.682 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:29.982 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:30.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.102+0000 7efe2684a640 10 monclient: get_monmap_and_config 2024-03-25T19:33:30.102 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.102+0000 7efe2684a640 10 monclient: build_initial_monmap 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.102+0000 7efe2684a640 10 monclient: monmap: 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:33:30.103826+0000 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:33:30.103826+0000 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:33:30.103 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:30.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.103+0000 7efe2684a640 1 Processor -- start 2024-03-25T19:33:30.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.103+0000 7efe2684a640 1 -- start start 2024-03-25T19:33:30.104 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.103+0000 7efe2684a640 10 monclient: init 2024-03-25T19:33:30.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.104+0000 7efe2684a640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:30.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.104+0000 7efe2684a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:30.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.104+0000 7efe2684a640 10 monclient(hunting): picked mon.noname-b con 0x7efe200a98f0 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:30.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.104+0000 7efe2684a640 10 monclient(hunting): picked mon.noname-a con 0x7efe2013f710 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:30.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.104+0000 7efe2684a640 10 monclient(hunting): picked mon.noname-c con 0x7efe201430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:30.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.104+0000 7efe2684a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe20142520 con 0x7efe2013f710 2024-03-25T19:33:30.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.104+0000 7efe2684a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe200b28a0 con 0x7efe201430c0 2024-03-25T19:33:30.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.104+0000 7efe2684a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe200b2a30 con 0x7efe200a98f0 2024-03-25T19:33:30.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.104+0000 7efe2684a640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:30.105 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.104+0000 7efe2684a640 10 monclient(hunting): authenticate will time out at 6247.464741s 2024-03-25T19:33:30.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.116+0000 7efe1ffff640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7efe200a98f0 0x7efe200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:37988/0 (socket says 172.21.15.123:37988) 2024-03-25T19:33:30.115 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.116+0000 7efe1ffff640 1 -- 172.21.15.123:0/3779552231 learned_addr learned my addr 172.21.15.123:0/3779552231 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:30.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.130+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025765848 0 0) 0x7efe200b2a30 con 0x7efe200a98f0 2024-03-25T19:33:30.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.130+0000 7efe1effd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:30.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.130+0000 7efe1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:30.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.130+0000 7efe1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:30.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.130+0000 7efe1effd640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:33:30.130 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.130+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe08003520 con 0x7efe200a98f0 2024-03-25T19:33:30.140 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.141+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696841947 0 0) 0x7efe20142520 con 0x7efe2013f710 2024-03-25T19:33:30.140 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.141+0000 7efe1effd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:30.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.141+0000 7efe1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:30.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.141+0000 7efe1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:30.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.141+0000 7efe1effd640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:33:30.141 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.141+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe200b2a30 con 0x7efe2013f710 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157205485 0 0) 0x7efe08003520 con 0x7efe200a98f0 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 >> v1:172.21.15.151:6789/0 conn(0x7efe201430c0 legacy=0x7efe2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 >> v1:172.21.15.123:6789/0 conn(0x7efe2013f710 legacy=0x7efe201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 1 monclient: found mon.noname-b 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.123:6790/0 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe200b44c0 con 0x7efe200a98f0 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 10 monclient: _finish_auth 0 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7efe10001690 con 0x7efe200a98f0 2024-03-25T19:33:30.145 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe2684a640 5 monclient: authenticate success, global_id 4724 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe2684a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe1effd640 10 monclient: dump: 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:30.146 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:30.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.145+0000 7efe2684a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:30.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.147+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 <== mon.2 v1:172.21.15.123:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe100017f0 con 0x7efe200a98f0 2024-03-25T19:33:30.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.147+0000 7efe1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:30.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.147+0000 7efe1effd640 1 -- 172.21.15.123:0/3779552231 <== mon.2 v1:172.21.15.123:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7efe10001c00 con 0x7efe200a98f0 2024-03-25T19:33:30.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.147+0000 7efe2684a640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:30.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.147+0000 7efe2684a640 4 set_mon_vals no callback set 2024-03-25T19:33:30.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.147+0000 7efe1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:30.147 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.147+0000 7efe1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.147+0000 7efe1effd640 10 monclient: dump: 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:30.148 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:30.150 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.148+0000 7efe2684a640 10 monclient: shutdown 2024-03-25T19:33:30.150 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.148+0000 7efe2684a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:30.150 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.148+0000 7efe2684a640 1 -- 172.21.15.123:0/3779552231 >> v1:172.21.15.123:6790/0 conn(0x7efe200a98f0 legacy=0x7efe200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.150 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.148+0000 7efe2684a640 1 -- 172.21.15.123:0/3779552231 shutdown_connections 2024-03-25T19:33:30.150 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.148+0000 7efe2684a640 1 -- 172.21.15.123:0/3779552231 >> 172.21.15.123:0/3779552231 conn(0x7efe200b7ea0 msgr2=0x7efe200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:30.150 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.148+0000 7efe2684a640 1 -- 172.21.15.123:0/3779552231 shutdown_connections 2024-03-25T19:33:30.150 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.148+0000 7efe2684a640 1 -- 172.21.15.123:0/3779552231 wait complete. 2024-03-25T19:33:30.150 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.149+0000 7efe2684a640 10 monclient: build_initial_monmap 2024-03-25T19:33:30.150 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.149+0000 7efe2684a640 10 monclient: monmap: 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.149+0000 7efe2684a640 1 librados: starting msgr at 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.149+0000 7efe2684a640 1 librados: starting objecter 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.149+0000 7efe2684a640 1 Processor -- start 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 1 -- start start 2024-03-25T19:33:30.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 1 librados: setting wanted keys 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 1 librados: calling monclient init 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 10 monclient: init 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 10 monclient(hunting): picked mon.noname-b con 0x7efe200a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 10 monclient(hunting): picked mon.noname-c con 0x7efe2013f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 10 monclient(hunting): picked mon.noname-a con 0x7efe201430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe200abd30 con 0x7efe201430c0 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe200abfa0 con 0x7efe200a98f0 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe201d77a0 con 0x7efe2013f710 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:30.152 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.150+0000 7efe2684a640 10 monclient(hunting): authenticate will time out at 6247.510584s 2024-03-25T19:33:30.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.168+0000 7efe1ffff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7efe200a98f0 0x7efe201d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:60140/0 (socket says 172.21.15.123:60140) 2024-03-25T19:33:30.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.168+0000 7efe1ffff640 1 -- 172.21.15.123:0/2604616512 learned_addr learned my addr 172.21.15.123:0/2604616512 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:30.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.169+0000 7efe1cff9640 10 client.?.objecter ms_handle_connect 0x7efe200a98f0 2024-03-25T19:33:30.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.169+0000 7efe1cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:30.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.174+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050590499 0 0) 0x7efe200abfa0 con 0x7efe200a98f0 2024-03-25T19:33:30.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.174+0000 7efe1cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:30.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.174+0000 7efe1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:30.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.174+0000 7efe1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:30.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.174+0000 7efe1cff9640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:33:30.174 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.174+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c004eb0 con 0x7efe200a98f0 2024-03-25T19:33:30.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.182+0000 7efe1cff9640 10 client.?.objecter ms_handle_connect 0x7efe2013f710 2024-03-25T19:33:30.182 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.182+0000 7efe1cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:30.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.186+0000 7efe1cff9640 10 client.?.objecter ms_handle_connect 0x7efe201430c0 2024-03-25T19:33:30.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.186+0000 7efe1cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:30.194 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.195+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171689234 0 0) 0x7efe200abd30 con 0x7efe201430c0 2024-03-25T19:33:30.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.195+0000 7efe1cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:30.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.195+0000 7efe1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:30.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.195+0000 7efe1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:30.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.195+0000 7efe1cff9640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:33:30.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.195+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe200abfa0 con 0x7efe201430c0 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.196+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3457137107 0 0) 0x7efe0c004eb0 con 0x7efe200a98f0 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.196+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe200abd30 con 0x7efe200a98f0 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.196+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7efe10001bc0 con 0x7efe200a98f0 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.196+0000 7efe1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.196+0000 7efe1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.196+0000 7efe1cff9640 10 monclient(hunting): dump: 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:30.196 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:30.197 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:30.197 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:30.197 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:30.197 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:30.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.196+0000 7efe1cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:30.201 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.201+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730326855 0 0) 0x7efe201d77a0 con 0x7efe2013f710 2024-03-25T19:33:30.201 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.201+0000 7efe1cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:30.201 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.201+0000 7efe1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:30.201 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.201+0000 7efe1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:30.201 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.201+0000 7efe1cff9640 10 monclient(hunting): my global_id is 4727 2024-03-25T19:33:30.201 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.201+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c004eb0 con 0x7efe2013f710 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.215+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.1 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2881477283 0 0) 0x7efe200abd30 con 0x7efe200a98f0 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.215+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 >> v1:172.21.15.123:6790/0 conn(0x7efe2013f710 legacy=0x7efe201d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.215+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 >> v1:172.21.15.123:6789/0 conn(0x7efe201430c0 legacy=0x7efe201385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.215+0000 7efe1cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.215+0000 7efe1cff9640 1 monclient: found mon.b 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.215+0000 7efe1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.215+0000 7efe1cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.215+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe201d83d0 con 0x7efe200a98f0 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe1cff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe1cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:30.216 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:30.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe2684a640 5 monclient: authenticate success, global_id 4714 2024-03-25T19:33:30.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe2684a640 10 monclient: _renew_subs 2024-03-25T19:33:30.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe2684a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:30.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe2684a640 1 -- 172.21.15.123:0/2604616512 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe201d41e0 con 0x7efe200a98f0 2024-03-25T19:33:30.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe2684a640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:30.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe2684a640 10 monclient: _renew_subs 2024-03-25T19:33:30.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe2684a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:30.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe2684a640 1 -- 172.21.15.123:0/2604616512 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe201d5110 con 0x7efe200a98f0 2024-03-25T19:33:30.217 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.216+0000 7efe2684a640 1 librados: init done 2024-03-25T19:33:30.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.217+0000 7efe2684a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:30.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.217+0000 7efe2684a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:30.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.217+0000 7efe2684a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:30.218 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.217+0000 7efe2684a640 1 -- 172.21.15.123:0/2604616512 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe201d7930 con 0x7efe200a98f0 2024-03-25T19:33:30.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.504+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.1 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe100011f0 con 0x7efe200a98f0 2024-03-25T19:33:30.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.504+0000 7efe1cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.504+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.1 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7efe10004cb0 con 0x7efe200a98f0 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.504+0000 7efe1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.504+0000 7efe1cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.504+0000 7efe1cff9640 10 monclient: dump: 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:30.505 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:30.506 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:30.506 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:30.506 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:30.506 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.504+0000 7efe1e7fc640 4 set_mon_vals no callback set 2024-03-25T19:33:30.506 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.505+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.1 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7efe1001ad00 con 0x7efe200a98f0 2024-03-25T19:33:30.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.505+0000 7efe1f7fe640 10 monclient: get_auth_request con 0x7efe0c06ddb0 auth_method 0 2024-03-25T19:33:30.507 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.506+0000 7efe1cff9640 10 client.4714.objecter ms_handle_connect 0x7efe0c06ddb0 2024-03-25T19:33:30.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.514+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.1 v1:172.21.15.151:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7efe10054970 con 0x7efe200a98f0 2024-03-25T19:33:30.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.514+0000 7efe1cff9640 10 client.4714.objecter ms_dispatch 0x7efe20000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:33:30.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.514+0000 7efe1cff9640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:33:30.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.514+0000 7efe1cff9640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:33:30.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.514+0000 7efe1cff9640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T19:33:30.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.514+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.1 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe100502b0 con 0x7efe200a98f0 2024-03-25T19:33:30.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.514+0000 7efe1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:30.514 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.514+0000 7efe1cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:30.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.750+0000 7efe2684a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:30.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.750+0000 7efe2684a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:30.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.750+0000 7efe2684a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.151:6789/0 2024-03-25T19:33:30.750 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.750+0000 7efe2684a640 1 -- 172.21.15.123:0/2604616512 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe201d8560 con 0x7efe200a98f0 2024-03-25T19:33:30.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.839+0000 7efe1cff9640 1 -- 172.21.15.123:0/2604616512 <== mon.1 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7efe100523b0 con 0x7efe200a98f0 2024-03-25T19:33:30.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.839+0000 7efe1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:30.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.839+0000 7efe1cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:30.839 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:33:30.839 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:30.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.841+0000 7efe2684a640 10 librados: watch_flush enter 2024-03-25T19:33:30.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 10 librados: watch_flush exit 2024-03-25T19:33:30.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T19:33:30.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 10 client.4714.objecter successfully canceled tick 2024-03-25T19:33:30.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 1 -- 172.21.15.123:0/2604616512 >> v1:172.21.15.151:6800/727684239 conn(0x7efe0c06ddb0 legacy=0x7efe0c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.842 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 10 monclient: shutdown 2024-03-25T19:33:30.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:30.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 1 -- 172.21.15.123:0/2604616512 >> v1:172.21.15.151:6789/0 conn(0x7efe200a98f0 legacy=0x7efe201d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 1 -- 172.21.15.123:0/2604616512 shutdown_connections 2024-03-25T19:33:30.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 1 -- 172.21.15.123:0/2604616512 >> 172.21.15.123:0/2604616512 conn(0x7efe200b7ea0 msgr2=0x7efe2013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:30.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 1 -- 172.21.15.123:0/2604616512 shutdown_connections 2024-03-25T19:33:30.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 1 -- 172.21.15.123:0/2604616512 wait complete. 2024-03-25T19:33:30.843 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:30.842+0000 7efe2684a640 1 librados: shutdown 2024-03-25T19:33:30.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:36.874 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:26.447 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:36.920 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:26.454 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:36.923 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:26.456 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:36.924 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:26.497 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:36.946 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:26.480 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:36.960 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:26.532 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:36.982 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:26.555 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:36.990 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:26.562 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:37.029 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:26.602 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:37.660 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:37.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.777+0000 7f59d1d23640 10 monclient: get_monmap_and_config 2024-03-25T19:33:37.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.777+0000 7f59d1d23640 10 monclient: build_initial_monmap 2024-03-25T19:33:37.777 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.777+0000 7f59d1d23640 10 monclient: monmap: 2024-03-25T19:33:37.777 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:37.777 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:37.777 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:33:37.778136+0000 2024-03-25T19:33:37.777 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:33:37.778136+0000 2024-03-25T19:33:37.777 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:37.777 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:37.778 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:37.778 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:33:37.778 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:33:37.778 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:37.779 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.778+0000 7f59d1d23640 1 Processor -- start 2024-03-25T19:33:37.779 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.778+0000 7f59d1d23640 1 -- start start 2024-03-25T19:33:37.779 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.778+0000 7f59d1d23640 10 monclient: init 2024-03-25T19:33:37.779 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.778+0000 7f59d1d23640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:37.779 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.778+0000 7f59d1d23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:37.779 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.778+0000 7f59d1d23640 10 monclient(hunting): picked mon.noname-c con 0x7f59cc0a98f0 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:37.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.778+0000 7f59d1d23640 10 monclient(hunting): picked mon.noname-b con 0x7f59cc13f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:37.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.778+0000 7f59d1d23640 10 monclient(hunting): picked mon.noname-a con 0x7f59cc1430c0 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:37.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.778+0000 7f59d1d23640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc142520 con 0x7f59cc1430c0 2024-03-25T19:33:37.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.779+0000 7f59d1d23640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc0b28a0 con 0x7f59cc0a98f0 2024-03-25T19:33:37.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.779+0000 7f59d1d23640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc0b2a30 con 0x7f59cc13f710 2024-03-25T19:33:37.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.779+0000 7f59d1d23640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:37.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.779+0000 7f59d1d23640 10 monclient(hunting): authenticate will time out at 6255.138914s 2024-03-25T19:33:37.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.779+0000 7f59cb7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f59cc0a98f0 0x7f59cc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:55864/0 (socket says 172.21.15.123:55864) 2024-03-25T19:33:37.780 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.779+0000 7f59cb7fe640 1 -- 172.21.15.123:0/3505213122 learned_addr learned my addr 172.21.15.123:0/3505213122 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:37.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.847+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084762255 0 0) 0x7f59cc0b2a30 con 0x7f59cc13f710 2024-03-25T19:33:37.846 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.847+0000 7f59ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:37.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.847+0000 7f59ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:37.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.847+0000 7f59ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:37.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.847+0000 7f59ca7fc640 10 monclient(hunting): my global_id is 4730 2024-03-25T19:33:37.847 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.847+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b4003520 con 0x7f59cc13f710 2024-03-25T19:33:37.852 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.853+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171900327 0 0) 0x7f59cc0b28a0 con 0x7f59cc0a98f0 2024-03-25T19:33:37.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.853+0000 7f59ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:37.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.853+0000 7f59ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:37.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.853+0000 7f59ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:37.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.853+0000 7f59ca7fc640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:33:37.853 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.853+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc0b2a30 con 0x7f59cc0a98f0 2024-03-25T19:33:37.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.855+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960846860 0 0) 0x7f59cc142520 con 0x7f59cc1430c0 2024-03-25T19:33:37.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.855+0000 7f59ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:37.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.855+0000 7f59ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:37.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.855+0000 7f59ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:37.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.855+0000 7f59ca7fc640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:33:37.855 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.855+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc0b28a0 con 0x7f59cc1430c0 2024-03-25T19:33:37.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 270654185 0 0) 0x7f59cc0b28a0 con 0x7f59cc1430c0 2024-03-25T19:33:37.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 >> v1:172.21.15.123:6790/0 conn(0x7f59cc13f710 legacy=0x7f59cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:37.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 >> v1:172.21.15.151:6789/0 conn(0x7f59cc0a98f0 legacy=0x7f59cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:37.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:37.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 1 monclient: found mon.noname-a 2024-03-25T19:33:37.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:37.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.123:6789/0 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc0b44c0 con 0x7f59cc1430c0 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f59c4001930 con 0x7f59cc1430c0 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59d1d23640 5 monclient: authenticate success, global_id 4704 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59d1d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59ca7fc640 10 monclient: dump: 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:37.878 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.879 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:37.879 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:37.879 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:37.879 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:37.879 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:37.879 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.877+0000 7f59d1d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:37.889 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.889+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 <== mon.0 v1:172.21.15.123:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59c4001b50 con 0x7f59cc1430c0 2024-03-25T19:33:37.889 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.889+0000 7f59ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:37.889 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.889+0000 7f59ca7fc640 1 -- 172.21.15.123:0/3505213122 <== mon.0 v1:172.21.15.123:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f59c4001ee0 con 0x7f59cc1430c0 2024-03-25T19:33:37.889 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.889+0000 7f59ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.889 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.889+0000 7f59ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:37.889 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.889+0000 7f59ca7fc640 10 monclient: dump: 2024-03-25T19:33:37.889 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:37.889 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:37.890 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:37.890 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:37.890 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.890 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:37.890 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:37.890 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:37.890 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:37.890 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:37.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.889+0000 7f59d1d23640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:37.890 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.889+0000 7f59d1d23640 4 set_mon_vals no callback set 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.890+0000 7f59d1d23640 10 monclient: shutdown 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.890+0000 7f59d1d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.890+0000 7f59d1d23640 1 -- 172.21.15.123:0/3505213122 >> v1:172.21.15.123:6789/0 conn(0x7f59cc1430c0 legacy=0x7f59cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.890+0000 7f59d1d23640 1 -- 172.21.15.123:0/3505213122 shutdown_connections 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.890+0000 7f59d1d23640 1 -- 172.21.15.123:0/3505213122 >> 172.21.15.123:0/3505213122 conn(0x7f59cc0b7ea0 msgr2=0x7f59cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.890+0000 7f59d1d23640 1 -- 172.21.15.123:0/3505213122 shutdown_connections 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.890+0000 7f59d1d23640 1 -- 172.21.15.123:0/3505213122 wait complete. 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.891+0000 7f59d1d23640 10 monclient: build_initial_monmap 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.891+0000 7f59d1d23640 10 monclient: monmap: 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:37.892 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.892+0000 7f59d1d23640 1 librados: starting msgr at 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.892+0000 7f59d1d23640 1 librados: starting objecter 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.892+0000 7f59d1d23640 1 Processor -- start 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.892+0000 7f59d1d23640 1 -- start start 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.892+0000 7f59d1d23640 1 librados: setting wanted keys 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.892+0000 7f59d1d23640 1 librados: calling monclient init 2024-03-25T19:33:37.893 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.892+0000 7f59d1d23640 10 monclient: init 2024-03-25T19:33:37.894 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.893+0000 7f59d1d23640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:37.894 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.893+0000 7f59d1d23640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:37.894 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.893+0000 7f59d1d23640 10 monclient(hunting): picked mon.noname-a con 0x7f59cc0a98f0 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:37.894 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.893+0000 7f59d1d23640 10 monclient(hunting): picked mon.noname-c con 0x7f59cc13f710 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:37.894 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.893+0000 7f59d1d23640 10 monclient(hunting): picked mon.noname-b con 0x7f59cc1430c0 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:37.894 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.893+0000 7f59d1d23640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc0abcf0 con 0x7f59cc0a98f0 2024-03-25T19:33:37.894 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.893+0000 7f59d1d23640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc1d51f0 con 0x7f59cc1430c0 2024-03-25T19:33:37.894 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.893+0000 7f59d1d23640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc1d83d0 con 0x7f59cc13f710 2024-03-25T19:33:37.894 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.893+0000 7f59d1d23640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:37.895 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.893+0000 7f59d1d23640 10 monclient(hunting): authenticate will time out at 6255.253284s 2024-03-25T19:33:37.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.901+0000 7f59cbfff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f59cc1430c0 0x7f59cc1e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:55892/0 (socket says 172.21.15.123:55892) 2024-03-25T19:33:37.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.901+0000 7f59cbfff640 1 -- 172.21.15.123:0/334187175 learned_addr learned my addr 172.21.15.123:0/334187175 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:37.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.903+0000 7f59b3fff640 10 client.?.objecter ms_handle_connect 0x7f59cc1430c0 2024-03-25T19:33:37.903 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.904+0000 7f59b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:37.909 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.909+0000 7f59b3fff640 10 client.?.objecter ms_handle_connect 0x7f59cc13f710 2024-03-25T19:33:37.909 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.909+0000 7f59b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:37.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.913+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349116414 0 0) 0x7f59cc1d51f0 con 0x7f59cc1430c0 2024-03-25T19:33:37.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.913+0000 7f59b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:37.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.913+0000 7f59b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:37.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.913+0000 7f59b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:37.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.913+0000 7f59b3fff640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:33:37.913 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.913+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b8004eb0 con 0x7f59cc1430c0 2024-03-25T19:33:37.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.916+0000 7f59b3fff640 10 client.?.objecter ms_handle_connect 0x7f59cc0a98f0 2024-03-25T19:33:37.916 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.916+0000 7f59b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:37.924 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.925+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197770596 0 0) 0x7f59cc1d83d0 con 0x7f59cc13f710 2024-03-25T19:33:37.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.925+0000 7f59b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:37.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.925+0000 7f59b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:37.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.925+0000 7f59b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:37.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.925+0000 7f59b3fff640 10 monclient(hunting): my global_id is 4733 2024-03-25T19:33:37.925 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.925+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc1d51f0 con 0x7f59cc13f710 2024-03-25T19:33:37.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.932+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125522422 0 0) 0x7f59cc0abcf0 con 0x7f59cc0a98f0 2024-03-25T19:33:37.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.932+0000 7f59b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:37.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.932+0000 7f59b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:37.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.932+0000 7f59b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:37.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.932+0000 7f59b3fff640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:33:37.932 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.932+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc1d83d0 con 0x7f59cc0a98f0 2024-03-25T19:33:37.936 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.937+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1995277730 0 0) 0x7f59b8004eb0 con 0x7f59cc1430c0 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.937+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59cc0abcf0 con 0x7f59cc1430c0 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.937+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f59c4001ea0 con 0x7f59cc1430c0 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.937+0000 7f59b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.937+0000 7f59b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.937+0000 7f59b3fff640 10 monclient(hunting): dump: 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:37.937 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:37.938 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.937+0000 7f59b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:37.946 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.947+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 374389211 0 0) 0x7f59cc1d51f0 con 0x7f59cc13f710 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.947+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59b8004eb0 con 0x7f59cc13f710 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.947+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f59c0001950 con 0x7f59cc13f710 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.947+0000 7f59b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.947+0000 7f59b3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.947+0000 7f59b3fff640 10 monclient(hunting): dump: 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:37.947 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:37.948 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:37.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.953+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 777318476 0 0) 0x7f59cc1d83d0 con 0x7f59cc0a98f0 2024-03-25T19:33:37.952 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.953+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59cc1d51f0 con 0x7f59cc0a98f0 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.953+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f59bc001930 con 0x7f59cc0a98f0 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.953+0000 7f59b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.953+0000 7f59b3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.953+0000 7f59b3fff640 10 monclient(hunting): dump: 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:37.953 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:37.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.956+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 915611279 0 0) 0x7f59b8004eb0 con 0x7f59cc13f710 2024-03-25T19:33:37.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.956+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 >> v1:172.21.15.151:6789/0 conn(0x7f59cc1430c0 legacy=0x7f59cc1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:37.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.956+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 >> v1:172.21.15.123:6789/0 conn(0x7f59cc0a98f0 legacy=0x7f59cc1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:37.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59b3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:33:37.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59b3fff640 1 monclient: found mon.c 2024-03-25T19:33:37.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:37.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:33:37.957 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc138e40 con 0x7f59cc13f710 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59b3fff640 10 monclient: _finish_auth 0 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59b3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59d1d23640 5 monclient: authenticate success, global_id 4733 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59d1d23640 10 monclient: _renew_subs 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59d1d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59d1d23640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59cc137f70 con 0x7f59cc13f710 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59d1d23640 10 client.4733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59d1d23640 10 monclient: _renew_subs 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59d1d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59d1d23640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc138270 con 0x7f59cc13f710 2024-03-25T19:33:37.958 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.957+0000 7f59d1d23640 1 librados: init done 2024-03-25T19:33:37.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.958+0000 7f59d1d23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:37.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.958+0000 7f59d1d23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:37.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.958+0000 7f59d1d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:33:37.959 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:37.958+0000 7f59d1d23640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59cc1d7600 con 0x7f59cc13f710 2024-03-25T19:33:38.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.271+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59c0001d60 con 0x7f59cc13f710 2024-03-25T19:33:38.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.271+0000 7f59b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:38.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.271+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f59c00035f0 con 0x7f59cc13f710 2024-03-25T19:33:38.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.271+0000 7f59b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:38.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.271+0000 7f59b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:38.272 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.271+0000 7f59b3fff640 10 monclient: dump: 2024-03-25T19:33:38.272 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:38.272 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:38.272 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:38.272 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:38.273 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:38.273 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:38.273 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:38.273 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:38.273 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:38.273 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:38.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.271+0000 7f59c9ffb640 4 set_mon_vals no callback set 2024-03-25T19:33:38.273 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.272+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f59c0019640 con 0x7f59cc13f710 2024-03-25T19:33:38.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.272+0000 7f59cb7fe640 10 monclient: get_auth_request con 0x7f59b806e160 auth_method 0 2024-03-25T19:33:38.274 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.273+0000 7f59b3fff640 10 client.4733.objecter ms_handle_connect 0x7f59b806e160 2024-03-25T19:33:38.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.281+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7f59c00533c0 con 0x7f59cc13f710 2024-03-25T19:33:38.282 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.281+0000 7f59b3fff640 10 client.4733.objecter ms_dispatch 0x7f59cc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:33:38.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.281+0000 7f59b3fff640 3 client.4733.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:33:38.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.281+0000 7f59b3fff640 3 client.4733.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:33:38.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.282+0000 7f59b3fff640 20 client.4733.objecter dump_active .. 0 homeless 2024-03-25T19:33:38.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.282+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59c004ed00 con 0x7f59cc13f710 2024-03-25T19:33:38.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.282+0000 7f59b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:38.283 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.282+0000 7f59b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:38.530 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.530+0000 7f59d1d23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:38.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.530+0000 7f59d1d23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:38.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.530+0000 7f59d1d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:33:38.531 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.530+0000 7f59d1d23640 1 -- 172.21.15.123:0/334187175 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59cc0627b0 con 0x7f59cc13f710 2024-03-25T19:33:38.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.555+0000 7f59b3fff640 1 -- 172.21.15.123:0/334187175 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7f59c0050e00 con 0x7f59cc13f710 2024-03-25T19:33:38.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.555+0000 7f59b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:38.556 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.555+0000 7f59b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:38.556 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:33:38.557 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:38.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 10 librados: watch_flush enter 2024-03-25T19:33:38.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 10 librados: watch_flush exit 2024-03-25T19:33:38.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 20 client.4733.objecter shutdown clearing up homeless session... 2024-03-25T19:33:38.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 10 client.4733.objecter successfully canceled tick 2024-03-25T19:33:38.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 1 -- 172.21.15.123:0/334187175 >> v1:172.21.15.151:6800/727684239 conn(0x7f59b806e160 legacy=0x7f59b8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:38.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 10 monclient: shutdown 2024-03-25T19:33:38.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:38.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 1 -- 172.21.15.123:0/334187175 >> v1:172.21.15.123:6790/0 conn(0x7f59cc13f710 legacy=0x7f59cc138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:38.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 1 -- 172.21.15.123:0/334187175 shutdown_connections 2024-03-25T19:33:38.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 1 -- 172.21.15.123:0/334187175 >> 172.21.15.123:0/334187175 conn(0x7f59cc0b7ea0 msgr2=0x7f59cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:38.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 1 -- 172.21.15.123:0/334187175 shutdown_connections 2024-03-25T19:33:38.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 1 -- 172.21.15.123:0/334187175 wait complete. 2024-03-25T19:33:38.560 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:38.558+0000 7f59d1d23640 1 librados: shutdown 2024-03-25T19:33:38.572 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:45.377 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:45.494 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.494+0000 7f9d56b4b640 10 monclient: get_monmap_and_config 2024-03-25T19:33:45.494 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.494+0000 7f9d56b4b640 10 monclient: build_initial_monmap 2024-03-25T19:33:45.494 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.494+0000 7f9d56b4b640 10 monclient: monmap: 2024-03-25T19:33:45.494 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:45.495 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:45.495 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:33:45.495724+0000 2024-03-25T19:33:45.495 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:33:45.495724+0000 2024-03-25T19:33:45.495 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:45.495 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:45.495 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:45.495 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:33:45.495 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:33:45.495 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:45.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.495+0000 7f9d56b4b640 1 Processor -- start 2024-03-25T19:33:45.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.495+0000 7f9d56b4b640 1 -- start start 2024-03-25T19:33:45.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.495+0000 7f9d56b4b640 10 monclient: init 2024-03-25T19:33:45.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.496+0000 7f9d56b4b640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:45.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.496+0000 7f9d56b4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:45.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.496+0000 7f9d56b4b640 10 monclient(hunting): picked mon.noname-c con 0x7f9d500b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:45.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.496+0000 7f9d56b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f9d500b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:45.496 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.496+0000 7f9d56b4b640 10 monclient(hunting): picked mon.noname-a con 0x7f9d5014be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:45.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.496+0000 7f9d56b4b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500af940 con 0x7f9d5014be90 2024-03-25T19:33:45.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.496+0000 7f9d56b4b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500b0a70 con 0x7f9d500b4d60 2024-03-25T19:33:45.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.496+0000 7f9d56b4b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500b0d50 con 0x7f9d500b1730 2024-03-25T19:33:45.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.496+0000 7f9d56b4b640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:45.497 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.496+0000 7f9d56b4b640 10 monclient(hunting): authenticate will time out at 6262.856213s 2024-03-25T19:33:45.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.504+0000 7f9d548e7640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f9d500b4d60 0x7f9d5013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:52498/0 (socket says 172.21.15.123:52498) 2024-03-25T19:33:45.504 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.504+0000 7f9d548e7640 1 -- 172.21.15.123:0/1911380509 learned_addr learned my addr 172.21.15.123:0/1911380509 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:45.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.510+0000 7f9d4f7fe640 1 -- 172.21.15.123:0/1911380509 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354848858 0 0) 0x7f9d500b0a70 con 0x7f9d500b4d60 2024-03-25T19:33:45.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.510+0000 7f9d4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:45.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.510+0000 7f9d4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:45.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.510+0000 7f9d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:45.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.510+0000 7f9d4f7fe640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:33:45.510 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.511+0000 7f9d4f7fe640 1 -- 172.21.15.123:0/1911380509 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d40003520 con 0x7f9d500b4d60 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 1 -- 172.21.15.123:0/1911380509 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928754006 0 0) 0x7f9d40003520 con 0x7f9d500b4d60 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 1 -- 172.21.15.123:0/1911380509 >> v1:172.21.15.123:6790/0 conn(0x7f9d500b1730 legacy=0x7f9d500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 1 -- 172.21.15.123:0/1911380509 >> v1:172.21.15.123:6789/0 conn(0x7f9d5014be90 legacy=0x7f9d5014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 1 -- 172.21.15.123:0/1911380509 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d500b1030 con 0x7f9d500b4d60 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:45.518 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 1 -- 172.21.15.123:0/1911380509 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9d44001690 con 0x7f9d500b4d60 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d4f7fe640 10 monclient: dump: 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d56b4b640 5 monclient: authenticate success, global_id 4723 2024-03-25T19:33:45.519 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.518+0000 7f9d56b4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:45.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.520+0000 7f9d4f7fe640 1 -- 172.21.15.123:0/1911380509 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d440017f0 con 0x7f9d500b4d60 2024-03-25T19:33:45.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.520+0000 7f9d4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:45.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.520+0000 7f9d4f7fe640 1 -- 172.21.15.123:0/1911380509 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9d44001c00 con 0x7f9d500b4d60 2024-03-25T19:33:45.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.520+0000 7f9d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:45.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.520+0000 7f9d4f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:45.520 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.520+0000 7f9d4f7fe640 10 monclient: dump: 2024-03-25T19:33:45.520 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:45.520 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:45.520 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:45.521 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:45.521 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:45.521 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:45.521 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:45.521 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:45.521 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:45.521 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:45.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.520+0000 7f9d56b4b640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:45.521 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.520+0000 7f9d56b4b640 4 set_mon_vals no callback set 2024-03-25T19:33:45.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.521+0000 7f9d56b4b640 10 monclient: shutdown 2024-03-25T19:33:45.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.521+0000 7f9d56b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:45.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.521+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1911380509 >> v1:172.21.15.151:6789/0 conn(0x7f9d500b4d60 legacy=0x7f9d5013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:45.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.521+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1911380509 shutdown_connections 2024-03-25T19:33:45.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.521+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1911380509 >> 172.21.15.123:0/1911380509 conn(0x7f9d500617a0 msgr2=0x7f9d50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:45.522 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.522+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1911380509 shutdown_connections 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.522+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1911380509 wait complete. 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.522+0000 7f9d56b4b640 10 monclient: build_initial_monmap 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.522+0000 7f9d56b4b640 10 monclient: monmap: 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.523+0000 7f9d56b4b640 1 librados: starting msgr at 2024-03-25T19:33:45.523 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.523+0000 7f9d56b4b640 1 librados: starting objecter 2024-03-25T19:33:45.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.523+0000 7f9d56b4b640 1 Processor -- start 2024-03-25T19:33:45.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.523+0000 7f9d56b4b640 1 -- start start 2024-03-25T19:33:45.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.523+0000 7f9d56b4b640 1 librados: setting wanted keys 2024-03-25T19:33:45.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.523+0000 7f9d56b4b640 1 librados: calling monclient init 2024-03-25T19:33:45.524 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.523+0000 7f9d56b4b640 10 monclient: init 2024-03-25T19:33:45.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.524+0000 7f9d56b4b640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:45.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.524+0000 7f9d56b4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:45.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.524+0000 7f9d56b4b640 10 monclient(hunting): picked mon.noname-a con 0x7f9d500b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:45.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.524+0000 7f9d56b4b640 10 monclient(hunting): picked mon.noname-c con 0x7f9d500b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:45.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.524+0000 7f9d56b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f9d5014be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:45.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.524+0000 7f9d56b4b640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d50141f40 con 0x7f9d500b1730 2024-03-25T19:33:45.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.524+0000 7f9d56b4b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d50142360 con 0x7f9d5014be90 2024-03-25T19:33:45.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.524+0000 7f9d56b4b640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d501424f0 con 0x7f9d500b4d60 2024-03-25T19:33:45.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.524+0000 7f9d56b4b640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:45.525 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.524+0000 7f9d56b4b640 10 monclient(hunting): authenticate will time out at 6262.884161s 2024-03-25T19:33:45.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.535+0000 7f9d548e7640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f9d500b1730 0x7f9d501405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:60158/0 (socket says 172.21.15.123:60158) 2024-03-25T19:33:45.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.535+0000 7f9d548e7640 1 -- 172.21.15.123:0/1255518338 learned_addr learned my addr 172.21.15.123:0/1255518338 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:45.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.536+0000 7f9d4d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d500b1730 2024-03-25T19:33:45.535 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.536+0000 7f9d4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:45.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.541+0000 7f9d4d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d500b4d60 2024-03-25T19:33:45.541 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.541+0000 7f9d4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:45.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.549+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965113674 0 0) 0x7f9d50141f40 con 0x7f9d500b1730 2024-03-25T19:33:45.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.549+0000 7f9d4d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:45.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.549+0000 7f9d4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:45.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.549+0000 7f9d4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:45.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.549+0000 7f9d4d7fa640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:33:45.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.549+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d34004eb0 con 0x7f9d500b1730 2024-03-25T19:33:45.549 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.550+0000 7f9d4d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d5014be90 2024-03-25T19:33:45.550 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.550+0000 7f9d4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:45.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.551+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350429360 0 0) 0x7f9d500b0d50 con 0x7f9d500b4d60 2024-03-25T19:33:45.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.551+0000 7f9d4d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:45.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.551+0000 7f9d4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:45.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.551+0000 7f9d4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:45.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.551+0000 7f9d4d7fa640 10 monclient(hunting): my global_id is 4739 2024-03-25T19:33:45.551 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.551+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d50141f40 con 0x7f9d500b4d60 2024-03-25T19:33:45.563 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.564+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2276137712 0 0) 0x7f9d34004eb0 con 0x7f9d500b1730 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.564+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d500b0d50 con 0x7f9d500b1730 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.564+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9d44001c00 con 0x7f9d500b1730 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.564+0000 7f9d4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.564+0000 7f9d4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.564+0000 7f9d4d7fa640 10 monclient(hunting): dump: 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:45.564 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:45.565 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:45.565 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:45.565 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.564+0000 7f9d4d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:45.566 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.567+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656318600 0 0) 0x7f9d50142360 con 0x7f9d5014be90 2024-03-25T19:33:45.566 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.567+0000 7f9d4d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:45.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.567+0000 7f9d4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:45.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.567+0000 7f9d4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:45.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.567+0000 7f9d4d7fa640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:33:45.567 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.567+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d34004eb0 con 0x7f9d5014be90 2024-03-25T19:33:45.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.573+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4176729300 0 0) 0x7f9d50141f40 con 0x7f9d500b4d60 2024-03-25T19:33:45.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.573+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d50142360 con 0x7f9d500b4d60 2024-03-25T19:33:45.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.573+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9d3c001700 con 0x7f9d500b4d60 2024-03-25T19:33:45.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.573+0000 7f9d4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:45.572 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.573+0000 7f9d4d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.573+0000 7f9d4d7fa640 10 monclient(hunting): dump: 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:45.573 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:45.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.0 v1:172.21.15.123:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2748015579 0 0) 0x7f9d500b0d50 con 0x7f9d500b1730 2024-03-25T19:33:45.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 >> v1:172.21.15.123:6790/0 conn(0x7f9d500b4d60 legacy=0x7f9d501d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:45.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 >> v1:172.21.15.151:6789/0 conn(0x7f9d5014be90 legacy=0x7f9d501e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:45.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:33:45.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 1 monclient: found mon.a 2024-03-25T19:33:45.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:45.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:45.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d50140d00 con 0x7f9d500b1730 2024-03-25T19:33:45.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:33:45.582 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d56b4b640 5 monclient: authenticate success, global_id 4713 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d56b4b640 10 monclient: _renew_subs 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d56b4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1255518338 --> v1:172.21.15.123:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d50138150 con 0x7f9d500b1730 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d56b4b640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d56b4b640 10 monclient: _renew_subs 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d56b4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1255518338 --> v1:172.21.15.123:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d501d8520 con 0x7f9d500b1730 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.581+0000 7f9d56b4b640 1 librados: init done 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.582+0000 7f9d56b4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.582+0000 7f9d56b4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.582+0000 7f9d56b4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:45.583 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.582+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1255518338 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d50138450 con 0x7f9d500b1730 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.613+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.0 v1:172.21.15.123:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d44004a90 con 0x7f9d500b1730 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.613+0000 7f9d4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.613+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.0 v1:172.21.15.123:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f9d44004e20 con 0x7f9d500b1730 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.613+0000 7f9d4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.613+0000 7f9d4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.613+0000 7f9d4d7fa640 10 monclient: dump: 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:45.613 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:45.614 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:45.614 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:45.614 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:45.614 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:45.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.613+0000 7f9d4effd640 4 set_mon_vals no callback set 2024-03-25T19:33:45.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.614+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.0 v1:172.21.15.123:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f9d44005040 con 0x7f9d500b1730 2024-03-25T19:33:45.614 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.615+0000 7f9d4ffff640 10 monclient: get_auth_request con 0x7f9d3406dfb0 auth_method 0 2024-03-25T19:33:45.615 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.615+0000 7f9d4d7fa640 10 client.4713.objecter ms_handle_connect 0x7f9d3406dfb0 2024-03-25T19:33:45.899 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.899+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.0 v1:172.21.15.123:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7f9d44054960 con 0x7f9d500b1730 2024-03-25T19:33:45.899 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.899+0000 7f9d4d7fa640 10 client.4713.objecter ms_dispatch 0x7f9d50000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:33:45.899 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.899+0000 7f9d4d7fa640 3 client.4713.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:33:45.899 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.899+0000 7f9d4d7fa640 3 client.4713.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:33:45.899 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.899+0000 7f9d4d7fa640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T19:33:45.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.899+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.0 v1:172.21.15.123:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d44050470 con 0x7f9d500b1730 2024-03-25T19:33:45.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.899+0000 7f9d4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:45.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:45.899+0000 7f9d4d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:46.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.112+0000 7f9d56b4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:46.111 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.112+0000 7f9d56b4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:46.112 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.112+0000 7f9d56b4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.123:6789/0 2024-03-25T19:33:46.112 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.112+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1255518338 --> v1:172.21.15.123:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d500627b0 con 0x7f9d500b1730 2024-03-25T19:33:46.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.157+0000 7f9d4d7fa640 1 -- 172.21.15.123:0/1255518338 <== mon.0 v1:172.21.15.123:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7f9d440523a0 con 0x7f9d500b1730 2024-03-25T19:33:46.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.157+0000 7f9d4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:46.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.157+0000 7f9d4d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:46.157 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:33:46.157 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 10 librados: watch_flush enter 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 10 librados: watch_flush exit 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 10 client.4713.objecter successfully canceled tick 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1255518338 >> v1:172.21.15.151:6800/727684239 conn(0x7f9d3406dfb0 legacy=0x7f9d34070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 10 monclient: shutdown 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1255518338 >> v1:172.21.15.123:6789/0 conn(0x7f9d500b1730 legacy=0x7f9d501405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1255518338 shutdown_connections 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1255518338 >> 172.21.15.123:0/1255518338 conn(0x7f9d500617a0 msgr2=0x7f9d500b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1255518338 shutdown_connections 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 1 -- 172.21.15.123:0/1255518338 wait complete. 2024-03-25T19:33:46.160 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:46.160+0000 7f9d56b4b640 1 librados: shutdown 2024-03-25T19:33:46.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:52.977 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:53.085 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.086+0000 7fbcabc8f640 10 monclient: get_monmap_and_config 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.086+0000 7fbcabc8f640 10 monclient: build_initial_monmap 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.086+0000 7fbcabc8f640 10 monclient: monmap: 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:33:53.087073+0000 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:33:53.087073+0000 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:33:53.086 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:53.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.086+0000 7fbcabc8f640 1 Processor -- start 2024-03-25T19:33:53.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.087+0000 7fbcabc8f640 1 -- start start 2024-03-25T19:33:53.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.087+0000 7fbcabc8f640 10 monclient: init 2024-03-25T19:33:53.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.087+0000 7fbcabc8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:53.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.087+0000 7fbcabc8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:53.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.087+0000 7fbcabc8f640 10 monclient(hunting): picked mon.noname-c con 0x7fbca40b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:53.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.087+0000 7fbcabc8f640 10 monclient(hunting): picked mon.noname-b con 0x7fbca40b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:53.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.088+0000 7fbcabc8f640 10 monclient(hunting): picked mon.noname-a con 0x7fbca414be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:53.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.088+0000 7fbcabc8f640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca40af940 con 0x7fbca414be90 2024-03-25T19:33:53.088 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.088+0000 7fbcabc8f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca40b0a70 con 0x7fbca40b4d60 2024-03-25T19:33:53.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.088+0000 7fbcabc8f640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca40b0d50 con 0x7fbca40b1730 2024-03-25T19:33:53.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.088+0000 7fbcabc8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:53.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.088+0000 7fbcabc8f640 10 monclient(hunting): authenticate will time out at 6270.447930s 2024-03-25T19:33:53.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.089+0000 7fbca9a2b640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fbca40b4d60 0x7fbca413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:57152/0 (socket says 172.21.15.123:57152) 2024-03-25T19:33:53.089 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.089+0000 7fbca9a2b640 1 -- 172.21.15.123:0/784660597 learned_addr learned my addr 172.21.15.123:0/784660597 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:53.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.149+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020100811 0 0) 0x7fbca40b0a70 con 0x7fbca40b4d60 2024-03-25T19:33:53.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.149+0000 7fbca8a29640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.149+0000 7fbca8a29640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.149+0000 7fbca8a29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.149+0000 7fbca8a29640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:33:53.149 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.149+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc90003520 con 0x7fbca40b4d60 2024-03-25T19:33:53.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.151+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476651834 0 0) 0x7fbca40af940 con 0x7fbca414be90 2024-03-25T19:33:53.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.151+0000 7fbca8a29640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.151+0000 7fbca8a29640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.151+0000 7fbca8a29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.151+0000 7fbca8a29640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:33:53.151 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.151+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca40b0a70 con 0x7fbca414be90 2024-03-25T19:33:53.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.153+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442191310 0 0) 0x7fbca40b0d50 con 0x7fbca40b1730 2024-03-25T19:33:53.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.153+0000 7fbca8a29640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.153+0000 7fbca8a29640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.153+0000 7fbca8a29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.153+0000 7fbca8a29640 10 monclient(hunting): my global_id is 4742 2024-03-25T19:33:53.153 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.153+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca40af940 con 0x7fbca40b1730 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.155+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1322811821 0 0) 0x7fbc90003520 con 0x7fbca40b4d60 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 >> v1:172.21.15.123:6790/0 conn(0x7fbca40b1730 legacy=0x7fbca40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 >> v1:172.21.15.123:6789/0 conn(0x7fbca414be90 legacy=0x7fbca414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 10 monclient: _finish_hunting 0 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 1 monclient: found mon.noname-c 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca40b1030 con 0x7fbca40b4d60 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 10 monclient: _finish_auth 0 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 10 monclient: _check_auth_tickets 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:53.156 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbca0001690 con 0x7fbca40b4d60 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbcabc8f640 5 monclient: authenticate success, global_id 4729 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbcabc8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbca8a29640 10 monclient: dump: 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:53.157 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:53.158 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.156+0000 7fbcabc8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:53.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.162+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbca00017f0 con 0x7fbca40b4d60 2024-03-25T19:33:53.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.162+0000 7fbca8a29640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:53.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.162+0000 7fbca8a29640 1 -- 172.21.15.123:0/784660597 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbca0001c00 con 0x7fbca40b4d60 2024-03-25T19:33:53.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.162+0000 7fbcabc8f640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:53.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.162+0000 7fbcabc8f640 4 set_mon_vals no callback set 2024-03-25T19:33:53.162 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.162+0000 7fbca8a29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.162+0000 7fbca8a29640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.162+0000 7fbca8a29640 10 monclient: dump: 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:53.163 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:53.165 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.163+0000 7fbcabc8f640 10 monclient: shutdown 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.163+0000 7fbcabc8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.163+0000 7fbcabc8f640 1 -- 172.21.15.123:0/784660597 >> v1:172.21.15.151:6789/0 conn(0x7fbca40b4d60 legacy=0x7fbca413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.163+0000 7fbcabc8f640 1 -- 172.21.15.123:0/784660597 shutdown_connections 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.163+0000 7fbcabc8f640 1 -- 172.21.15.123:0/784660597 >> 172.21.15.123:0/784660597 conn(0x7fbca40617a0 msgr2=0x7fbca4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.163+0000 7fbcabc8f640 1 -- 172.21.15.123:0/784660597 shutdown_connections 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.163+0000 7fbcabc8f640 1 -- 172.21.15.123:0/784660597 wait complete. 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.164+0000 7fbcabc8f640 10 monclient: build_initial_monmap 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.164+0000 7fbcabc8f640 10 monclient: monmap: 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:53.166 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.164+0000 7fbcabc8f640 1 librados: starting msgr at 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.164+0000 7fbcabc8f640 1 librados: starting objecter 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.164+0000 7fbcabc8f640 1 Processor -- start 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.164+0000 7fbcabc8f640 1 -- start start 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.164+0000 7fbcabc8f640 1 librados: setting wanted keys 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.164+0000 7fbcabc8f640 1 librados: calling monclient init 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.164+0000 7fbcabc8f640 10 monclient: init 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.165+0000 7fbcabc8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.165+0000 7fbcabc8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.165+0000 7fbcabc8f640 10 monclient(hunting): picked mon.noname-a con 0x7fbca40b1730 addr v1:172.21.15.123:6789/0 2024-03-25T19:33:53.167 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.165+0000 7fbcabc8f640 10 monclient(hunting): picked mon.noname-c con 0x7fbca40b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:33:53.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.165+0000 7fbcabc8f640 10 monclient(hunting): picked mon.noname-b con 0x7fbca414be90 addr v1:172.21.15.151:6789/0 2024-03-25T19:33:53.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.165+0000 7fbcabc8f640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca41d7730 con 0x7fbca40b1730 2024-03-25T19:33:53.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.165+0000 7fbcabc8f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca413bdf0 con 0x7fbca414be90 2024-03-25T19:33:53.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.165+0000 7fbcabc8f640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca4142100 con 0x7fbca40b4d60 2024-03-25T19:33:53.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.165+0000 7fbcabc8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:53.168 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.165+0000 7fbcabc8f640 10 monclient(hunting): authenticate will time out at 6270.525316s 2024-03-25T19:33:53.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.184+0000 7fbca922a640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7fbca40b4d60 0x7fbca413b6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:35556/0 (socket says 172.21.15.123:35556) 2024-03-25T19:33:53.184 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.184+0000 7fbca922a640 1 -- 172.21.15.123:0/3026844327 learned_addr learned my addr 172.21.15.123:0/3026844327 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:33:53.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.187+0000 7fbc9a7fc640 10 client.?.objecter ms_handle_connect 0x7fbca40b4d60 2024-03-25T19:33:53.186 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.187+0000 7fbc9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:53.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.193+0000 7fbc9a7fc640 10 client.?.objecter ms_handle_connect 0x7fbca414be90 2024-03-25T19:33:53.192 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.193+0000 7fbc9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:53.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.193+0000 7fbc9a7fc640 10 client.?.objecter ms_handle_connect 0x7fbca40b1730 2024-03-25T19:33:53.193 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.193+0000 7fbc9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:53.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.195+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145705442 0 0) 0x7fbca4142100 con 0x7fbca40b4d60 2024-03-25T19:33:53.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.195+0000 7fbc9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.195+0000 7fbc9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.195+0000 7fbc9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.195+0000 7fbc9a7fc640 10 monclient(hunting): my global_id is 4745 2024-03-25T19:33:53.195 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.195+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc88004eb0 con 0x7fbca40b4d60 2024-03-25T19:33:53.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.204+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918371818 0 0) 0x7fbca41d7730 con 0x7fbca40b1730 2024-03-25T19:33:53.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.204+0000 7fbc9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.204+0000 7fbc9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.203 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.204+0000 7fbc9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.204 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.204+0000 7fbc9a7fc640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:33:53.204 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.204+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca4142100 con 0x7fbca40b1730 2024-03-25T19:33:53.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.211+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027807367 0 0) 0x7fbca413bdf0 con 0x7fbca414be90 2024-03-25T19:33:53.210 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.211+0000 7fbc9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.211 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.211+0000 7fbc9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.211 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.211+0000 7fbc9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.211 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.211+0000 7fbc9a7fc640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:33:53.211 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.211+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca41d7730 con 0x7fbca414be90 2024-03-25T19:33:53.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.229+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3841190566 0 0) 0x7fbca41d7730 con 0x7fbca414be90 2024-03-25T19:33:53.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.229+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbca413bdf0 con 0x7fbca414be90 2024-03-25T19:33:53.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.229+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbc9c001bd0 con 0x7fbca414be90 2024-03-25T19:33:53.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.229+0000 7fbc9a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.229+0000 7fbc9a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:53.229 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.229+0000 7fbc9a7fc640 10 monclient(hunting): dump: 2024-03-25T19:33:53.229 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:53.229 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:53.229 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.230 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.230 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.230 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:53.230 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:53.230 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:53.230 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:53.230 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:53.230 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.229+0000 7fbc9a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.231+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2579896142 0 0) 0x7fbc88004eb0 con 0x7fbca40b4d60 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.231+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbca41d7730 con 0x7fbca40b4d60 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.231+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbc94001a10 con 0x7fbca40b4d60 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.231+0000 7fbc9a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.231+0000 7fbc9a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.231+0000 7fbc9a7fc640 10 monclient(hunting): dump: 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.231 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.232 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:53.232 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:53.232 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:53.232 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:53.232 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:53.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.238+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.0 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2724515544 0 0) 0x7fbca4142100 con 0x7fbca40b1730 2024-03-25T19:33:53.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.238+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.123:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc88004eb0 con 0x7fbca40b1730 2024-03-25T19:33:53.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.238+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.0 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbca0001b00 con 0x7fbca40b1730 2024-03-25T19:33:53.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.238+0000 7fbc9a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.238+0000 7fbc9a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:53.238 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.238+0000 7fbc9a7fc640 10 monclient(hunting): dump: 2024-03-25T19:33:53.238 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:53.238 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:53.238 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.238 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.239 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.239 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:53.239 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:53.239 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:53.239 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:53.239 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:53.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.251+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3367587235 0 0) 0x7fbca41d7730 con 0x7fbca40b4d60 2024-03-25T19:33:53.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 >> v1:172.21.15.151:6789/0 conn(0x7fbca414be90 legacy=0x7fbca41d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 >> v1:172.21.15.123:6789/0 conn(0x7fbca40b1730 legacy=0x7fbca41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbc9a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:53.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbc9a7fc640 1 monclient: found mon.c 2024-03-25T19:33:53.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbc9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:53.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbc9a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:33:53.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca4142290 con 0x7fbca40b4d60 2024-03-25T19:33:53.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbc9a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:33:53.252 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbc9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:53.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbc9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:53.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbcabc8f640 5 monclient: authenticate success, global_id 4745 2024-03-25T19:33:53.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbcabc8f640 10 monclient: _renew_subs 2024-03-25T19:33:53.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbcabc8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:33:53.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbcabc8f640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca41d8320 con 0x7fbca40b4d60 2024-03-25T19:33:53.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbcabc8f640 10 client.4745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:53.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbcabc8f640 10 monclient: _renew_subs 2024-03-25T19:33:53.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbcabc8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:33:53.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbcabc8f640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca41e33a0 con 0x7fbca40b4d60 2024-03-25T19:33:53.253 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.252+0000 7fbcabc8f640 1 librados: init done 2024-03-25T19:33:53.254 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.253+0000 7fbcabc8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:53.254 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.253+0000 7fbcabc8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:53.254 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.253+0000 7fbcabc8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:33:53.254 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.253+0000 7fbcabc8f640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca40627b0 con 0x7fbca40b4d60 2024-03-25T19:33:53.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.557+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc94001c70 con 0x7fbca40b4d60 2024-03-25T19:33:53.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.557+0000 7fbc9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:53.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.557+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7fbc94003a60 con 0x7fbca40b4d60 2024-03-25T19:33:53.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.557+0000 7fbc9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.557+0000 7fbc9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:53.557 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.557+0000 7fbc9a7fc640 10 monclient: dump: 2024-03-25T19:33:53.557 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:33:53.558 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:33:53.558 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.558 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:33:53.558 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.558 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:33:53.558 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:33:53.558 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:33:53.558 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:33:53.558 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:33:53.558 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.557+0000 7fbc9bfff640 4 set_mon_vals no callback set 2024-03-25T19:33:53.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.558+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7fbc94003c80 con 0x7fbca40b4d60 2024-03-25T19:33:53.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.559+0000 7fbca9a2b640 10 monclient: get_auth_request con 0x7fbc8806e160 auth_method 0 2024-03-25T19:33:53.559 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.559+0000 7fbc9a7fc640 10 client.4745.objecter ms_handle_connect 0x7fbc8806e160 2024-03-25T19:33:53.563 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.564+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7fbc94054810 con 0x7fbca40b4d60 2024-03-25T19:33:53.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.564+0000 7fbc9a7fc640 10 client.4745.objecter ms_dispatch 0x7fbca4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:33:53.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.564+0000 7fbc9a7fc640 3 client.4745.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:33:53.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.564+0000 7fbc9a7fc640 3 client.4745.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:33:53.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.564+0000 7fbc9a7fc640 20 client.4745.objecter dump_active .. 0 homeless 2024-03-25T19:33:53.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.564+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc94050320 con 0x7fbca40b4d60 2024-03-25T19:33:53.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.564+0000 7fbc9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:53.564 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.564+0000 7fbc9a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:53.815 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.816+0000 7fbcabc8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:53.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.816+0000 7fbcabc8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:53.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.816+0000 7fbcabc8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:33:53.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.816+0000 7fbcabc8f640 1 -- 172.21.15.123:0/3026844327 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbca4140170 con 0x7fbca40b4d60 2024-03-25T19:33:53.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.856+0000 7fbc9a7fc640 1 -- 172.21.15.123:0/3026844327 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7fbc94052250 con 0x7fbca40b4d60 2024-03-25T19:33:53.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.856+0000 7fbc9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:53.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.856+0000 7fbc9a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:53.856 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:33:53.857 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:53.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 10 librados: watch_flush enter 2024-03-25T19:33:53.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 10 librados: watch_flush exit 2024-03-25T19:33:53.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 20 client.4745.objecter shutdown clearing up homeless session... 2024-03-25T19:33:53.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 10 client.4745.objecter successfully canceled tick 2024-03-25T19:33:53.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 1 -- 172.21.15.123:0/3026844327 >> v1:172.21.15.151:6800/727684239 conn(0x7fbc8806e160 legacy=0x7fbc88070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 10 monclient: shutdown 2024-03-25T19:33:53.859 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:53.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 1 -- 172.21.15.123:0/3026844327 >> v1:172.21.15.123:6790/0 conn(0x7fbca40b4d60 legacy=0x7fbca413b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 1 -- 172.21.15.123:0/3026844327 shutdown_connections 2024-03-25T19:33:53.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 1 -- 172.21.15.123:0/3026844327 >> 172.21.15.123:0/3026844327 conn(0x7fbca40617a0 msgr2=0x7fbca40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:53.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 1 -- 172.21.15.123:0/3026844327 shutdown_connections 2024-03-25T19:33:53.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 1 -- 172.21.15.123:0/3026844327 wait complete. 2024-03-25T19:33:53.860 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:33:53.859+0000 7fbcabc8f640 1 librados: shutdown 2024-03-25T19:33:53.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:56.898 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:46.470 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:56.948 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:46.521 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:56.978 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:11:46.550 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:57.018 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:46.552 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:57.022 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:46.556 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:57.034 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:46.606 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:57.046 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:46.619 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:57.055 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:11:46.588 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:57.080 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:11:46.653 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:00.677 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:00.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.791+0000 7f1700f32640 10 monclient: get_monmap_and_config 2024-03-25T19:34:00.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.791+0000 7f1700f32640 10 monclient: build_initial_monmap 2024-03-25T19:34:00.791 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.791+0000 7f1700f32640 10 monclient: monmap: 2024-03-25T19:34:00.791 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:34:00.791 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:34:00.791 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:34:00.792616+0000 2024-03-25T19:34:00.791 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:34:00.792616+0000 2024-03-25T19:34:00.792 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:00.792 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:34:00.792 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:34:00.792 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-c 2024-03-25T19:34:00.792 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-b 2024-03-25T19:34:00.792 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:34:00.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.792+0000 7f1700f32640 1 Processor -- start 2024-03-25T19:34:00.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.792+0000 7f1700f32640 1 -- start start 2024-03-25T19:34:00.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.792+0000 7f1700f32640 10 monclient: init 2024-03-25T19:34:00.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.793+0000 7f1700f32640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:00.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.793+0000 7f1700f32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:00.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.793+0000 7f1700f32640 10 monclient(hunting): picked mon.noname-b con 0x7f16fc0b4d60 addr v1:172.21.15.123:6790/0 2024-03-25T19:34:00.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.793+0000 7f1700f32640 10 monclient(hunting): picked mon.noname-c con 0x7f16fc0b1730 addr v1:172.21.15.151:6789/0 2024-03-25T19:34:00.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.793+0000 7f1700f32640 10 monclient(hunting): picked mon.noname-a con 0x7f16fc14be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:34:00.793 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.793+0000 7f1700f32640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16fc0af940 con 0x7f16fc14be90 2024-03-25T19:34:00.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.793+0000 7f1700f32640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16fc0b0a70 con 0x7f16fc0b1730 2024-03-25T19:34:00.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.793+0000 7f1700f32640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16fc0b0d50 con 0x7f16fc0b4d60 2024-03-25T19:34:00.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.793+0000 7f1700f32640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:00.794 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.793+0000 7f1700f32640 10 monclient(hunting): authenticate will time out at 6278.153352s 2024-03-25T19:34:00.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.807+0000 7f16f3fff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f16fc0b1730 0x7f16fc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.123:52168/0 (socket says 172.21.15.123:52168) 2024-03-25T19:34:00.806 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.807+0000 7f16f3fff640 1 -- 172.21.15.123:0/3607548758 learned_addr learned my addr 172.21.15.123:0/3607548758 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:34:00.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.816+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889232064 0 0) 0x7f16fc0b0a70 con 0x7f16fc0b1730 2024-03-25T19:34:00.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.816+0000 7f16f9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.816+0000 7f16f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.816+0000 7f16f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.816+0000 7f16f9d9b640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:34:00.816 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.816+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e8003520 con 0x7f16fc0b1730 2024-03-25T19:34:00.826 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.827+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712399379 0 0) 0x7f16fc0b0d50 con 0x7f16fc0b4d60 2024-03-25T19:34:00.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.827+0000 7f16f9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.827+0000 7f16f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.827+0000 7f16f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.827+0000 7f16f9d9b640 10 monclient(hunting): my global_id is 4748 2024-03-25T19:34:00.827 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.827+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16fc0b0a70 con 0x7f16fc0b4d60 2024-03-25T19:34:00.828 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.829+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225650057 0 0) 0x7f16fc0af940 con 0x7f16fc14be90 2024-03-25T19:34:00.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.829+0000 7f16f9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.829+0000 7f16f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.829+0000 7f16f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.829+0000 7f16f9d9b640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:34:00.829 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.829+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16fc0b0d50 con 0x7f16fc14be90 2024-03-25T19:34:00.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2306031624 0 0) 0x7f16e8003520 con 0x7f16fc0b1730 2024-03-25T19:34:00.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 >> v1:172.21.15.123:6790/0 conn(0x7f16fc0b4d60 legacy=0x7f16fc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 >> v1:172.21.15.123:6789/0 conn(0x7f16fc14be90 legacy=0x7f16fc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:34:00.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 1 monclient: found mon.noname-c 2024-03-25T19:34:00.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:00.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.151:6789/0 2024-03-25T19:34:00.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc0b1030 con 0x7f16fc0b1730 2024-03-25T19:34:00.830 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f16e4001690 con 0x7f16fc0b1730 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f16f9d9b640 10 monclient: dump: 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:34:00.831 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:34:00.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f1700f32640 5 monclient: authenticate success, global_id 4735 2024-03-25T19:34:00.832 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.830+0000 7f1700f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:00.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.833+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 <== mon.1 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16e40017f0 con 0x7f16fc0b1730 2024-03-25T19:34:00.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.833+0000 7f16f9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:00.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.833+0000 7f16f9d9b640 1 -- 172.21.15.123:0/3607548758 <== mon.1 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f16e4001c00 con 0x7f16fc0b1730 2024-03-25T19:34:00.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.833+0000 7f1700f32640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:00.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.833+0000 7f1700f32640 4 set_mon_vals no callback set 2024-03-25T19:34:00.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.833+0000 7f16f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:00.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.833+0000 7f16f9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:00.833 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.833+0000 7f16f9d9b640 10 monclient: dump: 2024-03-25T19:34:00.833 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:34:00.833 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:34:00.834 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:34:00.834 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:34:00.834 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:00.834 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:34:00.834 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:34:00.834 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:34:00.834 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:34:00.834 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:34:00.835 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.834+0000 7f1700f32640 10 monclient: shutdown 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.834+0000 7f1700f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.834+0000 7f1700f32640 1 -- 172.21.15.123:0/3607548758 >> v1:172.21.15.151:6789/0 conn(0x7f16fc0b1730 legacy=0x7f16fc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.834+0000 7f1700f32640 1 -- 172.21.15.123:0/3607548758 shutdown_connections 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.834+0000 7f1700f32640 1 -- 172.21.15.123:0/3607548758 >> 172.21.15.123:0/3607548758 conn(0x7f16fc0617a0 msgr2=0x7f16fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.834+0000 7f1700f32640 1 -- 172.21.15.123:0/3607548758 shutdown_connections 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.834+0000 7f1700f32640 1 -- 172.21.15.123:0/3607548758 wait complete. 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.835+0000 7f1700f32640 10 monclient: build_initial_monmap 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.835+0000 7f1700f32640 10 monclient: monmap: 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:epoch 0 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:last_changed 0.000000 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:created 0.000000 2024-03-25T19:34:00.836 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 1 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.noname-a 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.noname-b 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.noname-c 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 1 librados: starting msgr at 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 1 librados: starting objecter 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 1 Processor -- start 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 1 -- start start 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 1 librados: setting wanted keys 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 1 librados: calling monclient init 2024-03-25T19:34:00.837 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 10 monclient: init 2024-03-25T19:34:00.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:00.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:00.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 10 monclient(hunting): picked mon.noname-c con 0x7f16fc0b1730 addr v1:172.21.15.123:6790/0 2024-03-25T19:34:00.838 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.836+0000 7f1700f32640 10 monclient(hunting): picked mon.noname-b con 0x7f16fc0b4d60 addr v1:172.21.15.151:6789/0 2024-03-25T19:34:00.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.837+0000 7f1700f32640 10 monclient(hunting): picked mon.noname-a con 0x7f16fc14be90 addr v1:172.21.15.123:6789/0 2024-03-25T19:34:00.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.837+0000 7f1700f32640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16fc1440c0 con 0x7f16fc14be90 2024-03-25T19:34:00.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.837+0000 7f1700f32640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16fc13f020 con 0x7f16fc0b4d60 2024-03-25T19:34:00.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.837+0000 7f1700f32640 1 -- --> v1:172.21.15.123:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16fc136d30 con 0x7f16fc0b1730 2024-03-25T19:34:00.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.837+0000 7f1700f32640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:00.839 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.837+0000 7f1700f32640 10 monclient(hunting): authenticate will time out at 6278.196943s 2024-03-25T19:34:00.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.856+0000 7f16fa59c640 1 --1- >> v1:172.21.15.123:6790/0 conn(0x7f16fc0b1730 0x7f16fc1439b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.123:6790/0 says I am v1:172.21.15.123:58832/0 (socket says 172.21.15.123:58832) 2024-03-25T19:34:00.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.856+0000 7f16fa59c640 1 -- 172.21.15.123:0/3388813755 learned_addr learned my addr 172.21.15.123:0/3388813755 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-03-25T19:34:00.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.857+0000 7f16f2ffd640 10 client.?.objecter ms_handle_connect 0x7f16fc0b1730 2024-03-25T19:34:00.856 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.857+0000 7f16f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:00.857 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.858+0000 7f16f2ffd640 10 client.?.objecter ms_handle_connect 0x7f16fc0b4d60 2024-03-25T19:34:00.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.858+0000 7f16f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:00.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.859+0000 7f16f2ffd640 10 client.?.objecter ms_handle_connect 0x7f16fc14be90 2024-03-25T19:34:00.858 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.859+0000 7f16f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:00.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.863+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.2 v1:172.21.15.123:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231146810 0 0) 0x7f16fc136d30 con 0x7f16fc0b1730 2024-03-25T19:34:00.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.863+0000 7f16f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.863+0000 7f16f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.863+0000 7f16f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.863+0000 7f16f2ffd640 10 monclient(hunting): my global_id is 4751 2024-03-25T19:34:00.863 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.863+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 --> v1:172.21.15.123:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16dc004eb0 con 0x7f16fc0b1730 2024-03-25T19:34:00.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797861126 0 0) 0x7f16fc13f020 con 0x7f16fc0b4d60 2024-03-25T19:34:00.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 10 monclient(hunting): my global_id is 4738 2024-03-25T19:34:00.877 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16fc136d30 con 0x7f16fc0b4d60 2024-03-25T19:34:00.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.0 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572412196 0 0) 0x7f16fc1440c0 con 0x7f16fc14be90 2024-03-25T19:34:00.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:34:00.878 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.877+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16fc13f020 con 0x7f16fc14be90 2024-03-25T19:34:00.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.881+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.2 v1:172.21.15.123:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2543146072 0 0) 0x7f16dc004eb0 con 0x7f16fc0b1730 2024-03-25T19:34:00.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.881+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 --> v1:172.21.15.123:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16fc1440c0 con 0x7f16fc0b1730 2024-03-25T19:34:00.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.881+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.2 v1:172.21.15.123:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f16ec001a30 con 0x7f16fc0b1730 2024-03-25T19:34:00.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.881+0000 7f16f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:00.881 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.881+0000 7f16f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.881+0000 7f16f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:34:00.882 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.881+0000 7f16f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:00.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.886+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1395472607 0 0) 0x7f16fc136d30 con 0x7f16fc0b4d60 2024-03-25T19:34:00.885 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.886+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16dc004eb0 con 0x7f16fc0b4d60 2024-03-25T19:34:00.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.892+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f16e4001b00 con 0x7f16fc0b4d60 2024-03-25T19:34:00.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.892+0000 7f16f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:00.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.892+0000 7f16f2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:00.891 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.892+0000 7f16f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:34:00.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.2 v1:172.21.15.123:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3618043667 0 0) 0x7f16fc1440c0 con 0x7f16fc0b1730 2024-03-25T19:34:00.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 >> v1:172.21.15.151:6789/0 conn(0x7f16fc0b4d60 legacy=0x7f16fc1d7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 >> v1:172.21.15.123:6789/0 conn(0x7f16fc14be90 legacy=0x7f16fc1d8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:00.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 1 monclient: found mon.c 2024-03-25T19:34:00.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:00.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:34:00.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 --> v1:172.21.15.123:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc136ec0 con 0x7f16fc0b1730 2024-03-25T19:34:00.900 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f16f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f1700f32640 5 monclient: authenticate success, global_id 4751 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f1700f32640 10 monclient: _renew_subs 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f1700f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f1700f32640 1 -- 172.21.15.123:0/3388813755 --> v1:172.21.15.123:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16fc1e33d0 con 0x7f16fc0b1730 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f1700f32640 10 client.4751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f1700f32640 10 monclient: _renew_subs 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f1700f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f1700f32640 1 -- 172.21.15.123:0/3388813755 --> v1:172.21.15.123:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16fc1e3740 con 0x7f16fc0b1730 2024-03-25T19:34:00.901 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.900+0000 7f1700f32640 1 librados: init done 2024-03-25T19:34:00.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.901+0000 7f1700f32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:00.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.901+0000 7f1700f32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:00.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.901+0000 7f1700f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:34:00.902 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:00.901+0000 7f1700f32640 1 -- 172.21.15.123:0/3388813755 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16fc1401f0 con 0x7f16fc0b1730 2024-03-25T19:34:01.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.197+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.2 v1:172.21.15.123:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16ec001b90 con 0x7f16fc0b1730 2024-03-25T19:34:01.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.197+0000 7f16f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:01.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.197+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.2 v1:172.21.15.123:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1881862815 0 0) 0x7f16ec0038c0 con 0x7f16fc0b1730 2024-03-25T19:34:01.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.197+0000 7f16f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:01.197 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.197+0000 7f16f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.197+0000 7f16f2ffd640 10 monclient: dump: 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:epoch 1 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:fsid 6494055c-b81c-4348-a74d-41946ff3e6ce 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:last_changed 2024-03-25T19:21:37.390888+0000 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:created 2024-03-25T19:21:37.390888+0000 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:election_strategy: 3 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:0: v1:172.21.15.123:6789/0 mon.a 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:1: v1:172.21.15.151:6789/0 mon.b 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:2: v1:172.21.15.123:6790/0 mon.c 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr: 2024-03-25T19:34:01.198 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.197+0000 7f16f959a640 4 set_mon_vals no callback set 2024-03-25T19:34:01.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.198+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.2 v1:172.21.15.123:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 966621404 0 0) 0x7f16ec003ae0 con 0x7f16fc0b1730 2024-03-25T19:34:01.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.199+0000 7f16f3fff640 10 monclient: get_auth_request con 0x7f16dc06e0c0 auth_method 0 2024-03-25T19:34:01.199 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.199+0000 7f16f2ffd640 10 client.4751.objecter ms_handle_connect 0x7f16dc06e0c0 2024-03-25T19:34:01.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.205+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.2 v1:172.21.15.123:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1926672122 0 0) 0x7f16ec054810 con 0x7f16fc0b1730 2024-03-25T19:34:01.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.205+0000 7f16f2ffd640 10 client.4751.objecter ms_dispatch 0x7f16fc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:34:01.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.205+0000 7f16f2ffd640 3 client.4751.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:34:01.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.205+0000 7f16f2ffd640 3 client.4751.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:34:01.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.205+0000 7f16f2ffd640 20 client.4751.objecter dump_active .. 0 homeless 2024-03-25T19:34:01.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.205+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.2 v1:172.21.15.123:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16ec050320 con 0x7f16fc0b1730 2024-03-25T19:34:01.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.205+0000 7f16f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:01.205 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.205+0000 7f16f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:01.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.434+0000 7f1700f32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:01.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.434+0000 7f1700f32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:01.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.434+0000 7f1700f32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.123:6790/0 2024-03-25T19:34:01.434 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.434+0000 7f1700f32640 1 -- 172.21.15.123:0/3388813755 --> v1:172.21.15.123:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16fc0627b0 con 0x7f16fc0b1730 2024-03-25T19:34:01.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.478+0000 7f16f2ffd640 1 -- 172.21.15.123:0/3388813755 <== mon.2 v1:172.21.15.123:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1705341153) 0x7f16ec052250 con 0x7f16fc0b1730 2024-03-25T19:34:01.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.478+0000 7f16f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:01.478 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.478+0000 7f16f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:01.478 INFO:teuthology.misc.health.smithi123.stdout: 2024-03-25T19:34:01.478 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":11,"fsid":"6494055c-b81c-4348-a74d-41946ff3e6ce","created":"2024-03-25T19:22:32.652664+0000","modified":"2024-03-25T19:32:36.590591+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:32:36.590591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c17b60d-1ca6-4aea-93f8-fb79f8a41266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6ebf8ea1-3898-4326-8acc-ea0056f287b0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"96cf3100-b688-4a78-87ef-06b8a6806ac3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f0fa10c2-a2a8-4392-9d03-a1b93522a284","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1738fb73-735d-4c91-9442-f6f6aa9c3a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"27d55d50-50bb-48a2-8870-d038cb804e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9b913f26-999c-4664-b1cf-49b52b0b1cda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e7c5e0e-b326-4567-b3a5-7950666afc0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:01.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 10 librados: watch_flush enter 2024-03-25T19:34:01.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 10 librados: watch_flush exit 2024-03-25T19:34:01.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 20 client.4751.objecter shutdown clearing up homeless session... 2024-03-25T19:34:01.480 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 10 client.4751.objecter successfully canceled tick 2024-03-25T19:34:01.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 1 -- 172.21.15.123:0/3388813755 >> v1:172.21.15.151:6800/727684239 conn(0x7f16dc06e0c0 legacy=0x7f16dc070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:01.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 10 monclient: shutdown 2024-03-25T19:34:01.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:01.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 1 -- 172.21.15.123:0/3388813755 >> v1:172.21.15.123:6790/0 conn(0x7f16fc0b1730 legacy=0x7f16fc1439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:01.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 1 -- 172.21.15.123:0/3388813755 shutdown_connections 2024-03-25T19:34:01.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 1 -- 172.21.15.123:0/3388813755 >> 172.21.15.123:0/3388813755 conn(0x7f16fc0617a0 msgr2=0x7f16fc13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:01.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 1 -- 172.21.15.123:0/3388813755 shutdown_connections 2024-03-25T19:34:01.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 1 -- 172.21.15.123:0/3388813755 wait complete. 2024-03-25T19:34:01.481 INFO:teuthology.misc.health.smithi123.stderr:2024-03-25T19:34:01.480+0000 7f1700f32640 1 librados: shutdown 2024-03-25T19:34:01.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:01.492 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:34:01.494 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:34:01.494 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:34:01.494 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:03.585 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:11:25.430 35426== 2024-03-25T19:34:03.586 INFO:tasks.ceph.osd.0.smithi123.stderr:==00:00:11:25.430 35426== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:03.623 INFO:tasks.ceph.osd.0.smithi123.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:07.596 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:34:07.597 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:34:07.597 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:34:07.597 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:09.485 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:11:31.330 35425== 2024-03-25T19:34:09.485 INFO:tasks.ceph.osd.1.smithi123.stderr:==00:00:11:31.330 35425== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:09.524 INFO:tasks.ceph.osd.1.smithi123.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:13.699 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:34:13.699 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:34:13.700 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:34:13.700 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:15.635 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:11:37.485 35424== 2024-03-25T19:34:15.635 INFO:tasks.ceph.osd.2.smithi123.stderr:==00:00:11:37.485 35424== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:15.676 INFO:tasks.ceph.osd.2.smithi123.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:16.968 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:12:06.540 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:16.978 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:12:06.550 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:17.001 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:12:06.574 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:17.019 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:12:06.591 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:17.020 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:12:06.554 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:17.028 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:12:06.562 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:17.033 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:12:06.606 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:17.049 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:12:06.583 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:17.051 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:12:06.623 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:19.802 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:34:19.802 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:34:19.802 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:34:19.803 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:21.564 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:11:43.424 35423== 2024-03-25T19:34:21.564 INFO:tasks.ceph.osd.3.smithi123.stderr:==00:00:11:43.424 35423== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:21.604 INFO:tasks.ceph.osd.3.smithi123.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:25.905 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:34:25.905 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:34:25.905 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:34:25.905 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:27.825 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:11:49.669 35401== 2024-03-25T19:34:27.825 INFO:tasks.ceph.osd.4.smithi151.stderr:==00:00:11:49.669 35401== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:27.863 INFO:tasks.ceph.osd.4.smithi151.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:32.007 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:34:32.008 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:34:32.008 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:34:32.008 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:33.996 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:11:55.854 35399== 2024-03-25T19:34:33.996 INFO:tasks.ceph.osd.5.smithi151.stderr:==00:00:11:55.854 35399== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:34.037 INFO:tasks.ceph.osd.5.smithi151.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:37.127 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:12:26.699 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:37.135 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:12:26.708 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:37.159 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:12:26.732 35106== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:37.179 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:12:26.712 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:37.182 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:12:26.716 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:37.199 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:12:26.732 34855== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:37.205 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:12:26.776 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:37.206 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:12:26.779 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:37.227 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:12:26.800 35113== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:38.109 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:34:38.110 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:34:38.110 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:34:38.110 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:39.856 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:12:01.708 35400== 2024-03-25T19:34:39.856 INFO:tasks.ceph.osd.6.smithi151.stderr:==00:00:12:01.708 35400== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:39.893 INFO:tasks.ceph.osd.6.smithi151.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:44.212 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:34:44.212 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:34:44.213 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:34:44.213 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:45.762 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:12:07.599 35402== 2024-03-25T19:34:45.763 INFO:tasks.ceph.osd.7.smithi151.stderr:==00:00:12:07.599 35402== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:45.794 INFO:tasks.ceph.osd.7.smithi151.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:50.315 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:34:50.315 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:34:50.315 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:34:50.316 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:34:50.316 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:50.343 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:34:50.343 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:34:50.343 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:50.344 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:34:50.344+0000 7f3278e3f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34853) UID: 0 2024-03-25T19:34:50.344 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-25T19:34:50.344+0000 7f3278e3f640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T19:34:50.373 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:34:50.373 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:34:50.374 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:34:50.374 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:50.399 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-03-25T19:34:50.389+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35078) UID: 0 2024-03-25T19:34:50.400 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-03-25T19:34:50.390+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:34:51.000 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:12:40.573 35106== 2024-03-25T19:34:51.000 INFO:tasks.ceph.mon.a.smithi123.stderr:==00:00:12:40.573 35106== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:51.010 INFO:tasks.ceph.mon.a.smithi123.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:56.475 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:34:56.475 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:34:56.476 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:34:56.476 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:56.491 INFO:tasks.ceph.mon.c.smithi123.stderr:2024-03-25T19:34:56.481+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35080) UID: 0 2024-03-25T19:34:56.492 INFO:tasks.ceph.mon.c.smithi123.stderr:2024-03-25T19:34:56.482+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:35:46.827 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:13:36.399 35113== 2024-03-25T19:35:46.827 INFO:tasks.ceph.mon.c.smithi123.stderr:==00:00:13:36.399 35113== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:35:46.836 INFO:tasks.ceph.mon.c.smithi123.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:35:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:35:51.388 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:35:51.389 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:35:51.389 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:35:51.389 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:35:51.405 INFO:tasks.ceph.mon.b.smithi151.stderr:2024-03-25T19:35:51.394+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34851) UID: 0 2024-03-25T19:35:51.406 INFO:tasks.ceph.mon.b.smithi151.stderr:2024-03-25T19:35:51.396+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T19:35:52.063 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:13:41.596 34855== 2024-03-25T19:35:52.063 INFO:tasks.ceph.mon.b.smithi151.stderr:==00:00:13:41.596 34855== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:35:52.072 INFO:tasks.ceph.mon.b.smithi151.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:35:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:35:57.492 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:35:57.492 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:35:57.492 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:35:57.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:35:57.543 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T19:35:57.543 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:35:57.612 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T19:35:57.613 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:35:57.680 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T19:35:57.681 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:35:57.737 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T19:35:57.737 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:35:57.796 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T19:35:57.796 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:35:57.888 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T19:35:57.888 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:35:57.962 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T19:35:57.962 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:35:58.029 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T19:35:58.029 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:35:58.086 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:35:58.087 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312/data/mon.a.tgz 2024-03-25T19:35:58.088 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-03-25T19:35:58.103 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.bXmQwzfIC0 2024-03-25T19:35:58.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.bXmQwzfIC0 2024-03-25T19:35:58.207 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.bXmQwzfIC0 2024-03-25T19:35:58.292 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.bXmQwzfIC0 is 134KB 2024-03-25T19:35:58.344 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.bXmQwzfIC0 2024-03-25T19:35:58.358 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312/data/mon.c.tgz 2024-03-25T19:35:58.359 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-03-25T19:35:58.412 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.k0jmAcIGLD 2024-03-25T19:35:58.413 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.k0jmAcIGLD 2024-03-25T19:35:58.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.k0jmAcIGLD 2024-03-25T19:35:58.585 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.k0jmAcIGLD is 135KB 2024-03-25T19:35:58.636 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.k0jmAcIGLD 2024-03-25T19:35:58.650 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312/data/mon.b.tgz 2024-03-25T19:35:58.650 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-03-25T19:35:58.668 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.kF3rcZj3D4 2024-03-25T19:35:58.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.kF3rcZj3D4 2024-03-25T19:35:58.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.kF3rcZj3D4 2024-03-25T19:35:58.864 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.kF3rcZj3D4 is 135KB 2024-03-25T19:35:58.916 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.kF3rcZj3D4 2024-03-25T19:35:58.930 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:35:58.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:35:58.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:35:59.258 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:35:59.471 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:35:59.473 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312/remote/smithi123/crash 2024-03-25T19:35:59.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:35:59.501 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312/remote/smithi151/crash 2024-03-25T19:35:59.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:35:59.529 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:35:59.529 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:35:59.545 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:35:59.615 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:35:59.615 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:35:59.615 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:35:59.615 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:35:59.615 INFO:teuthology.orchestra.run.smithi151.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:35:59.615 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:35:59.615 INFO:teuthology.orchestra.run.smithi151.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:35:59.615 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:35:59.615 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35009.log 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35209.log 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/valgrind/osd.4.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.35009.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.35209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35169.log 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35009.log.gz 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35300.log 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35209.log.gz 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.35169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35129.log 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.35300.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35169.log.gz 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.35129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35049.log 2024-03-25T19:35:59.616 INFO:teuthology.orchestra.run.smithi151.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35300.log.gz 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35129.log.gz 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.35049.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.7.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35049.log.gz 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35089.log 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.35089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35260.log 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35089.log.gz 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.35260.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr: 87.7%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.35260.log.gz 2024-03-25T19:35:59.617 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:35:59.619 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:35:59.619 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39348.log 2024-03-25T19:35:59.619 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39153.log 2024-03-25T19:35:59.619 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40011.log 2024-03-25T19:35:59.619 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41382.log 2024-03-25T19:35:59.619 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39348.log.gz 2024-03-25T19:35:59.619 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40011.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39153.log.gz 2024-03-25T19:35:59.619 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40756.log 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39660.log 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41382.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40011.log.gz 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38956.log 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41382.log.gz 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40245.log 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40756.log.gz 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5/var/log/ceph/ceph-client.admin.38956.log: --verbose -- /var/log/ceph/ceph-client.admin.41107.log 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39660.log.gz 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37533.log 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38956.log.gz 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40245.log.gz 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39504.log 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42124.log 2024-03-25T19:35:59.620 INFO:teuthology.orchestra.run.smithi123.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41107.log.gz 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39504.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37533.log.gz 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42124.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39504.log.gz 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42124.log.gz 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr: 72.1%/var/log/ceph/valgrind/osd.1.log: -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42085.log 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:35:59.621 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39231.log 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40873.log 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42085.log.gz 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40951.log 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39231.log.gz 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41146.log 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40873.log.gz 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41538.log 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41146.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40951.log.gz 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41029.log 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41811.log 2024-03-25T19:35:59.622 INFO:teuthology.orchestra.run.smithi123.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41146.log.gz 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41694.log 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41538.log.gz 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41811.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41029.log.gz 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40364.log 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41694.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41811.log.gz 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39270.log 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41694.log.gz 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39621.log 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40364.log.gz 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35449.log 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39621.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40637.log 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39270.log.gz 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40403.log 86.7% 2024-03-25T19:35:59.623 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39621.log.gz 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35449.log: /var/log/ceph/ceph-client.admin.40637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38995.log 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.35449.log.gz 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41577.log 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40637.log.gz 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41304.log 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40403.log.gz 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41968.log 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38995.log.gz 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41265.log 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41577.log.gz 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41968.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41304.log.gz 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39972.log 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41968.log.gz 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40167.log 2024-03-25T19:35:59.624 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41265.log.gz 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39972.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40834.log 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41616.log 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39972.log.gz 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40834.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40167.log.gz 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr: 86.7%/var/log/ceph/ceph-client.admin.41616.log: -- replaced with /var/log/ceph/ceph-client.admin.40834.log.gz 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41185.log 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40559.log 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41616.log.gz 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40481.log 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39894.log 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41185.log.gz 2024-03-25T19:35:59.625 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40481.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40559.log.gz 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38746.log 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39894.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40481.log.gz 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41068.log 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40598.log 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38746.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39894.log.gz 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41226.log 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40598.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38746.log.gzgzip 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35604.log 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41068.log.gz 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40598.log.gz 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35604.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41226.log.gz 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40089.log 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42007.log 2024-03-25T19:35:59.626 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41499.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35604.log.gz 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42007.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41929.log 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40089.log.gz 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40676.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42007.log.gz 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41929.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40442.log 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41929.log.gz/var/log/ceph/ceph-osd.3.log: 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 86.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz /var/log/ceph/ceph-client.admin.40128.log 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39933.log 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39036.log 2024-03-25T19:35:59.627 INFO:teuthology.orchestra.run.smithi123.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40442.log.gz 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40286.log 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40128.log.gz 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39192.log/var/log/ceph/ceph-client.admin.39036.log: 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr: 87.0%/var/log/ceph/ceph-client.admin.40286.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.39933.log.gz 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40206.log 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39192.log: gzip -5 --verbose 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40286.log.gz 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr: -- /var/log/ceph/ceph-client.admin.42163.log 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40206.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39192.log.gz 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39777.log 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42163.log: gzip 86.5% -5 --verbose -- /var/log/ceph/ceph-client.admin.42202.log 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40206.log.gz 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39777.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42163.log.gz 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42202.log: gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39777.log.gz 2024-03-25T19:35:59.628 INFO:teuthology.orchestra.run.smithi123.stderr: -- /var/log/ceph/ceph-client.admin.42241.log 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39387.log 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42202.log.gz 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40520.log 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39387.log: gzip 86.9% -5 --verbose -- /var/log/ceph/ceph-client.admin.39543.log 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42241.log.gz 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39387.log.gz 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36475.log 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr:gzip/var/log/ceph/ceph-client.admin.39543.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41343.log 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40520.log.gz 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39816.log 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39543.log.gz 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5/var/log/ceph/ceph-client.admin.41343.log: --verbose 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36475.log.gz 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr: -- /var/log/ceph/ceph-client.admin.40990.log 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39816.log: gzip -5 --verbose 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41343.log.gz 2024-03-25T19:35:59.629 INFO:teuthology.orchestra.run.smithi123.stderr: -- /var/log/ceph/ceph-client.admin.39582.log 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40990.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39816.log.gz 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39582.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40990.log.gz 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39465.log 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39582.log.gz 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39855.log 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35120.log 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40325.log 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39465.log.gz 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35120.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39855.log.gzgzip 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41733.log 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40325.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41733.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35120.log.gz 2024-03-25T19:35:59.630 INFO:teuthology.orchestra.run.smithi123.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40325.log.gz 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41733.log.gz 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39426.log 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41890.log/var/log/ceph/ceph.log: 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39426.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41890.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39426.log.gz 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29266.log 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-osd.2.log: 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41890.log.gz 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35422.log 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40050.log 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph.tmp-client.admin.29266.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29266.log.gz 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41772.log 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35422.log.gz/var/log/ceph/ceph-client.admin.40050.log: 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40050.log.gz 2024-03-25T19:35:59.631 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39309.log 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40795.log 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41772.log: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39309.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41772.log.gz 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39309.log.gz 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40717.log 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40717.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40795.log.gz 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42046.log 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- 86.9% /var/log/ceph/ceph-client.admin.39738.log 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40717.log.gz 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41655.log: /var/log/ceph/ceph-client.admin.42046.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gz 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42046.log.gz 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41421.log 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39699.log 2024-03-25T19:35:59.632 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41421.log: 86.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40912.log 2024-03-25T19:35:59.633 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39738.log.gz 2024-03-25T19:35:59.633 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39699.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41421.log.gz 2024-03-25T19:35:59.633 INFO:teuthology.orchestra.run.smithi123.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39699.log.gz 2024-03-25T19:35:59.633 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39075.log 2024-03-25T19:35:59.633 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40912.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:35:59.633 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39075.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40912.log.gz 2024-03-25T19:35:59.633 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mon.c.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39075.log.gz 2024-03-25T19:35:59.659 INFO:teuthology.orchestra.run.smithi151.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:36:00.431 INFO:teuthology.orchestra.run.smithi123.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:36:00.439 INFO:teuthology.orchestra.run.smithi123.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:36:00.500 INFO:teuthology.orchestra.run.smithi151.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:36:00.520 INFO:teuthology.orchestra.run.smithi151.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:36:00.539 INFO:teuthology.orchestra.run.smithi151.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:36:00.722 INFO:teuthology.orchestra.run.smithi123.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:36:00.727 INFO:teuthology.orchestra.run.smithi123.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:36:00.817 INFO:teuthology.orchestra.run.smithi151.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:36:04.618 INFO:teuthology.orchestra.run.smithi123.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:36:04.807 INFO:teuthology.orchestra.run.smithi151.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:36:04.808 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-25T19:36:04.809 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m5.219s 2024-03-25T19:36:04.809 INFO:teuthology.orchestra.run.smithi151.stderr:user 0m8.905s 2024-03-25T19:36:04.809 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m0.332s 2024-03-25T19:36:04.886 INFO:teuthology.orchestra.run.smithi123.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:36:04.888 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-03-25T19:36:04.888 INFO:teuthology.orchestra.run.smithi123.stderr:real 0m5.331s 2024-03-25T19:36:04.888 INFO:teuthology.orchestra.run.smithi123.stderr:user 0m13.888s 2024-03-25T19:36:04.888 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m0.514s 2024-03-25T19:36:04.888 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:36:04.889 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312/remote/smithi123/log 2024-03-25T19:36:04.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:36:07.269 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312/remote/smithi151/log 2024-03-25T19:36:07.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:36:08.748 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:36:08.958 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=39a73249c95b42b9a48cbdc9e6dd5cd3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:36:08.962 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:36:08.972 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:36:09.058 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:36:09.059 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:36:09.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:36:09.151 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:36:09.160 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:36:09.160 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:36:09.163 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:36:09.177 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:36:09.180 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:36:09.182 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:36:09.182 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-03-25T19:36:09.182 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 -223us[ -214us] +/- 80ms 2024-03-25T19:36:09.182 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -335us[ -326us] +/- 50ms 2024-03-25T19:36:09.182 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +746us[ +746us] +/- 85ms 2024-03-25T19:36:09.182 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:36:09.185 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:36:09.185 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-25T19:36:09.185 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 -202us[ -202us] +/- 80ms 2024-03-25T19:36:09.186 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -312us[ -313us] +/- 50ms 2024-03-25T19:36:09.186 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +779us[ +778us] +/- 85ms 2024-03-25T19:36:09.186 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:36:09.186 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:36:09.195 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:36:09.196 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:36:09.212 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:36:09.295 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:36:09.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:36:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:36:09.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:36:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:36:09.429 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:36:09.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:36:09.470 INFO:teuthology.task.internal:Duration was 6041.485721 seconds 2024-03-25T19:36:09.471 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:36:09.487 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:36:09.487 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:36:09.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:36:09.532 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:36:09.534 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:36:09.895 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:36:09.895 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-03-25T19:36:09.896 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:36:09.916 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-03-25T19:36:09.916 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:36:09.942 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:36:09.942 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:36:09.959 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:36:10.011 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:36:10.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:36:10.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:36:10.155 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:36:10.165 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:36:10.165 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:36:10.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:36:10.219 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:36:10.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:36:10.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:36:10.257 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-03-25T19:36:10.285 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-03-25T19:36:10.331 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:36:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:36:10.345 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:36:10.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:36:10.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:36:10.396 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:36:10.396 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312/remote/smithi123 2024-03-25T19:36:10.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:36:10.431 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312/remote/smithi151 2024-03-25T19:36:10.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:36:10.467 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:36:10.467 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:36:10.469 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:36:10.516 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:36:10.527 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:36:10.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:36:10.546 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:36:10.546 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:36:10.548 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:36:10.560 INFO:teuthology.orchestra.run.smithi123.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:36 /home/ubuntu/cephtest 2024-03-25T19:36:10.571 INFO:teuthology.orchestra.run.smithi151.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:36 /home/ubuntu/cephtest 2024-03-25T19:36:10.572 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:36:10.582 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:36:10.613 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:36:10.640 DEBUG:teuthology.nuke:shortname: smithi123 2024-03-25T19:36:10.640 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:36:10.663 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:12.706049', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSVD7vP2KGEGEkXf8ZFIHPD1QPwQEUOKMkbsOPXii1QG6h3uRKB2zvn8oiBAn+z+4YhpYIGHVnkHe0aNLvWdS0='} 2024-03-25T19:36:10.666 DEBUG:teuthology.nuke:shortname: smithi151 2024-03-25T19:36:10.666 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:36:10.689 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:12.704349', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUmqGh/fvs4G4Grt3FCYf2UIBDkVXoq+MdCOmetXyIl4A1NWCCJbh5kiJQcDb5hCwB04Fl8GbhBTVyZKB+7Tdg='} 2024-03-25T19:36:10.709 INFO:teuthology.orchestra.console.smithi123:Power off 2024-03-25T19:36:10.709 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:36:10.732 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2024-03-25T19:36:10.733 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:36:10.750 INFO:teuthology.orchestra.console.smithi151:Power off 2024-03-25T19:36:10.750 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:36:10.761 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-25T19:36:10.774 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-03-25T19:36:10.774 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:36:10.796 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-25T19:36:14.763 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:36:14.797 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:36:14.873 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-25T19:36:14.906 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-25T19:36:18.875 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:36:18.907 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:36:18.985 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-25T19:36:19.017 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-03-25T19:36:19.118 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-03-25T19:36:22.990 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:36:23.100 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-03-25T19:36:23.201 INFO:teuthology.orchestra.console.smithi123:Power off completed 2024-03-25T19:36:23.305 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6041.48572063446 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=39a73249c95b42b9a48cbdc9e6dd5cd3 status: fail success: false 2024-03-25T19:36:23.306 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:36:23.388 INFO:teuthology.run:FAIL